Automation Framework Ebook

Software Testing : Automation Framework’s eBook from www.OneStopTesting.
com
Software Testing Automation

Framework
Free Study Material from
www.OneStopTesting.com
World’s Largest Portal on Software Testing Information & Jobs -

http://www.OneStopTesting.com
Join Software Testing Community at
http://groups.yahoo.com/group/OneStopTesting/
Over 5,000 Testing Interview Questions at
http://www.CoolInterview.com
Software Testing : Automation Framework’s eBook from www.OneStopTesting.com
Automation Framework
The Software Testing Automation Framework (STAF) is an open source project
that enables users to create cross-platform, distributed software test environments.
The project is licensed using the Common Public License (CPL). STAF includes a
number of services that provide specific functionality. The most prominent of these is
the test execution engine called STAX. Other services provide cron, file system,
inter-process communication, e-mail, and HTML support, among others.
STAF was originally developed for internal use by IBM in the 1990's. IBM supports
users of STAF through extensive online documentation and user forums.

Keyword-driven Testing
Keyword driven testing is a software testing technique that separates much of the
programming work of test automation from the actual test design. This allows tests
to be developed earlier and makes the tests easier to maintain. Some key concepts
in keyword driven testing include:
• Keywords, which are typically base level and describe generalized UI

operations such as "click", "enter", "select"
• Business templates which are typically high level such as "login", "enter
transaction"
• Action Words, or short "Actions", which can be both base level and high level
and in their most general form allow earlier defined key words to be used to
define higher level action words
Keyword driven testing is a very powerful tool helping organizations to do more

automated testing earlier in the testing process and making it easier to maintain
tests over time. As with any complex undertaking, there are "success factors" that
can determine whether or not a testing effort will be successful. This paper will
outline key success factors for keyword driven testing including base requirements,
the vision for automation, success factors for automation, and how to measure
success.
Contents:
Overview
Base Requirements
The Framework
Methodology
Vision for Automation
Success Factors for Automation
How to Measure Success
Conclusion

Overview
Keyword-driven testing or Table-driven testing is a Software testing
methodology as part of the Test automation discipline that separates the
programming work from the actual Test script so that the test scripts can be
developed without knowledge of programming. As a result the test scripts can be
maintained with only minor updates, even when the application or testing requires
significant changes.
Base Requirements
There are numerous requirements that consider to be "base requirements" for
success with keyword driven testing. These include:
• Test development and automation must be fully separated – It is very

important to separate test development from test automation. The two
disciplines require very different skills. Fundamentally, testers are not and
should not be programmers. Testers must be adept at defining test cases
independent of the underlying technology to implement them. Individuals who
are skilled technically, the "automation people" (automation engineers), will
implement the action words and then test them.
• Test cases must have a clear and differentiated scope – It is important that
test cases have a clearly differentiated scope and that they not deviate from
that scope.
• The tests must be written at the right level of abstraction – Tests must be
written at the right level of abstraction such as the higher business level,
lower user interface level, or both. It is also important that test tools provide
this level of flexibility.

The Framework
The implementation of this methodology is framework dependent. This framework
requires the development of data tables and keywords, independent of the test
automation tool used to execute them and the test script code that "drives" the
application-under-test and the data. In a keyword-driven test, the functionality of
the application-under-test is documented in a table as well as in step-by-step
instructions for each test.
If we were to map out the actions we perform with the mouse when we test our
Windows Calculator functions by hand, we could create the following table. The
"Window" column contains the name of the application window where we're
performing the mouse action .
Methodology
The keyword-driven testing methodology divides test creation into two stages:
• Planning Stage
Analyzing the application and determining which objects and operations are
used by the set of business processes that need to be tested.
Determining which operations require customized keywords to provide
additional functionality, to achieve business-level clarity, and/or to maximize
efficiency and maintainability.
• Implementation Stage
Building a collection of references that uniquely identify objects, sometimes

known as an "object repository", and ensuring that all such references have
clear names that follow any predetermined naming conventions. (This is
primarily for test automation.)
Developing and documenting business-level keywords in function libraries.
Creating function libraries involves developing customized functions for the
application that needs to be tested.
Although this methodology requires more planning and a longer initial time-
investment than going directly to the test creation stage and recording your steps,
this methodology makes the test creation and test maintenance stages more efficient
and keeps the structure of individual tests more readable and easier to modify.

Vision for Automation

It is also important to have a clear vision for automation. Such a "vision" should
include things such as:
• Having a good methodology – It is important to have a good integrated

methodology for testing and automation that places testers in the driver’s
seat. It is also important to employ the best technology that supports the
methodology, maximizes flexibility, minimizes technical efforts, and
maximizes maintainability.
• Have the right tool – Any tool that is employed should be specifically designed
for keyword based testing. It should be flexible enough to allow for the right
mix of high and low level testing. It should allow the testers to quickly build
keyword tests, without difficulty. It should also not be overly complicated for
automation engineers to use when implementing the automation.
• Succeed in the three "success factors for automation" – There are three
critical success factors for automation that the vision should account for. They
are:
1. test design
2. automation solution
3. organization

Success Factors for Automation
Test Design
Test design is more important than the automation technology. Design is the most
underestimated part of testing. It is my belief that test design, not automation or a
tool, is the single most important factor for automation success. To understand more
about test design see these previous articles:
• Key Principles of Test Design

• The First Holy Grail of Test Design
• The Second Holy Grail of Test Design
• The Third Holy Grail of Test Design
Comprehensive Automation Architecture
• Scope, assumptions, risks

• Methods, best practices
• Tools, technologies, architecture
• Stake holders, including roles and processes for input and approvals
• …and more…
The "right" team must also be assembled. This team should include:
• Test management who is responsible for managing the test process.

• Test development who is responsible for production of tests. Test
development should include test leads, test developers, end users, subject
matter experts, and business analysts.
• Automation engineering who is responsible for creating the automation
scheme for automatic execution. Members of this team include a lead
engineer as well as one or more automation support engineers.
• Support functions, providing methods, techniques, know how, training, tools,
and environments.
For the team there should be a clear division of tasks and responsibilities as well as
well defined processes for decision making and communication.
How to Measure Success

With any major undertaking, it is important to define and measure "success". There
are two important areas of measurement for success – progress and quality.

Progress
You should measure test development against the test development plan. If goals
are not reached, act quickly to find the problems. Is the subject matter clear? Are
stake holders providing enough input? Is it clear what to test (overall, per module)?
Is the team right (enough, right skill set mix)?
You should measure automation and look at things such as implemented keywords
(actions) and interface definitions (defined interface dialogs, pages, etc).
You should measure test execution looking at things such as how many modules are
executed and how many executed correctly (without errors)?
Quality
Some of the key quality metrics include:
• Coverage of system and requirements

• Assessments by peers, test leads, and by stake holders (recommended)
• Effectiveness
o Are you finding bugs?
o Are you missing bugs?
o Can you find known bugs (or seeded bugs)?
o After the system is released, what bugs still come up? You should
consider calculating the "Defect Detection Percentage" (Dorothy
Graham, Mark Fewster)
• Mine your bug base for additional insights
Conclusion
It is important to understand that keywords are not magic, but they can serve you
well. What is more important is to take the effort seriously and "do it right". Doing it
right means that test design is essential, both global test design and the design of
individual test cases. Automation should be done but it should not dominate the
process. Automation should flow from the overall strategy, methodology, and
architecture. It is also very important to pay attention to organization – the process,
team, and project environment.

Data-driven Testing
Automated tests play back a recorded (or programmed) sequence of user actions
that cover a certain area of the tested application. To get larger coverage, you can
perform tests with different input data. Suppose, for example, you test an input form
of an application. After you have recorded a script, it contains only values that you
entered during the recording. Most likely, data you have specified does not cause
errors in the application, but other data may cause them. Data-driven testing allows
you to test the form with a different set of input values to be sure that the
application works as expected.
Contents:
Introduction
Data Driven Automation Frameworks
Data Driven Scripts
TestComplete's Support for Data-Driven Testing

Introduction
Data-driven testing means that scripts read data from an external storage site, for
example, from a file or database, rather than use values hard-coded in the script.
Such a separation makes tests logically simpler. A script containing several sets of
hard-coded values can be rather difficult to modify. For instance, if you need more
input data, you will have to modify the script code.
Normally, the storages hold both input and verifying data. A data-driven test
includes the following operations performed in a loop:
1. Retrieving input data from storage.

2. Entering data in an application form.
3. Verifying the results.
4. Continuing with the next set of input data.
TestComplete includes a number of features to perform these steps. You can select
the functionality that best suits your needs.
Operations
Following are the operations of Data-driven Testing:
Retrieving input data from storage
It is convenient to keep data for tests in special storages that support sequential
access to a set of data, for example, script arrays, text files, database tables, Excel
sheets, and so on. You can access data in these storages using scripting objects
provided by TestComplete. To make the access even easier, TestComplete includes
special features for extracting data from CSV (comma-separated values) files,
database tables and Excel sheets.
Entering data in an application form
To enter data into application forms and controls, you can either simulate
keystrokes, or store data to a property of the desired form or control. The way you
simulate the input depends on many factors. For example, it depends on whether
your application is compiled as an Open Application.

Verifying results
The easiest way to check whether your test actions were performed successfully is to
store the check value along with the set of input data and then compare the output
with these values. If the output is an image or a file, you can use TestComplete’s
features to compare the images or the files' and object properties.

Data Driven Automation Frameworks

Over the past several years there have been numerous articles done on various
approaches to test automation. Anyone who has read a fair, unbiased sampling of
these knows that we cannot and must not expect pure capture and replay of test
scripts to be successful for the life of a product. We will find nothing but frustration
there.
Sometimes this manifesto is hard to explain to people who have not yet performed
significant test automation with these capture\replay tools. But it usually takes less
than a week, often less than a day, to hear the most repeated phrase: "It worked
when I recorded it, but now it fails when I play it back!"
Data Driven Scripts

Data driven scripts are those application-specific scripts captured or manually coded
in the automation tool’s proprietary language and then modified to accommodate
variable data. Variables will be used for key application input fields and program
selections allowing the script to drive the application with external data supplied by
the calling routine or the shell that invoked the test script.
Variable Data, Hard Coded Component Identification

These data driven scripts often still contain the hard coded and sometimes very
fragile recognition strings for the window components they navigate. When this is
the case, the scripts are easily broken when an application change or revision occurs.
And when these scripts start breaking, we are not necessarily talking about just a
few. We are sometimes talking about a great many, if not all the scripts, for the
entire application.
Figure 1 is an example of activating a server-side image map link in a web

application with an automation tool scripting language:
This particular scenario of clicking on the image map might exist thousands of times
throughout all the scripts that test this application. The preceding example identifies
the image by the title given to the document and the index of the image on the
page. The hard coded image identification might work successfully all the way
through the production release of that version of the application. Consequently,
testers responsible for the automated test scripts may gain a false sense of security
and satisfaction with these results.

However, the next release cycle may find some or all of these scripts broken because
either the title of the document or the index of the image has changed. Sometimes,
with the right tools, this might not be too hard to fix. Sometimes, no matter what
tools, it will be frustratingly difficult.
Remember, we are potentially talking about thousands of broken lines of test script
code. And this is just one particular change. Where there is one, there will likely be
others.
Highly Technical or Duplicate Test Designs

Another common feature of data driven scripts is that virtually all of the test design
effort for the application is developed in the scripting language of the automation
tool. Either that, or it is duplicated in both manual and automated script versions.
This means that everyone involved with automated test development or automated
test execution for the application must likely become proficient in the environment
and programming language of the automation tool.
Findings:
A test automation framework relying on data driven scripts is definitely the easiest
and quickest to implement if you have and keep the technical staff to maintain it. But
it is the hardest of the data driven approaches to maintain and perpetuate and very
often leads to long-term failure.

TestComplete's Support for Data-Driven Testing

Since DDT itself is very straightforward, TestComplete’s direct support for the
technique is likewise quite straightforward. If you examine the TestComplete online
help, under “data-driven testing”, you’ll find a fairly long help topic with links to
external pages on the subject, but also links to five script routines. It’s worth noting
that while these methods are listed with the DDT topics, they’re not the only routines
you’ll need to perform DDT, but for now we’ll look at those five. They are
GetCSVCount, GetCSVItem, CallMethod, SetValue and GetValue. The first two
methods are ones that you’ll use quite a bit in DDT, the other three will see less
frequent use unless you are getting into some fairly robust DDT scripting.
GetCSVCount and GetCSVItem work with comma-separated strings in order to

provide an easy way to get to the values contained within the strings. This is called
“parsing”. A comma-separated value (CSV) string might look something like this:
02,c,”John”,”Q”,”Public”,13,”Product 02”,”1234 Anywhere St.”,”San Antonio”,”TX”
You can see that each element of this string is separated by a comma. Each
individual element of the string is a called a “field” or a “value”. The GetCSVCount
and GetCSVItem methods work with strings like this. If we passed this string in to
the GetCSVCount function, we’d get a return value of 10. There are ten discreet
values in this CSV string. In addition, if we wanted to the get the customer’s first
name, we could use GetCSVItem to do that.
Here’s an important thing to keep in mind: GetCSVItem is what’s known as “zero-

based”. That means that the first item in the CSV string is considered number 0
rather than number 1. Likewise, if we wanted the last item, it would be item number
9, not number 10. Thus, if we wanted the customer’s first name, we’d have to ask
for item number 2, not number 3, thus:
varCustFirstName = GetCSVItem(‘02,c,”John”,”Q”,”Public”,13,”Product 02”,”1234

Anywhere St.”,”San Antonio”,”TX”’,
So, with these two functions, we’re able to easily retrieve values from CSV files,
something that is quite useful if our external DDT data is in CSV text files or
spreadsheets. Returning again to our customer/orders example, we might see the
following. (This is pseudo-code only.)

function ReadExternalData(aFileName)
{
var lsLine
var lsCustomerName, lsAddr, lsCity, lsRegion, lsPostalCode
openFile(aFileName)
while not EOF(aFileName)
{
lsLine = readLine(aFileName)
if (GetCSVCount(lsLine) == 5
{
lsCustomerName = GetCSVItem(lsLine, 0)
lsAddr = GetCSVItem(lsLine, 1)
lsCity = GetCSVItem(lsLine, 2)
lsRegion = GetCSVItem(lsLine, 3)
lsPostalCode = GetCSVItem(lsLine, 4)
EnterCustomerRecord(lsCustomerName, lsAddr, lsCity, lsRegion, lsPostalCode)

}
}
closeFile(aFileName)
}
function EnterCustomerRecord(aCustomerName, aAddr, aCity, aRegion,
aPostalCode)
{
var p, w
p = Sys.OrdersProcess
w = p.MainForm
w.Activate()
w.ToolButton5.Click(6, 7)
w = p.OrderFrm
w.Activate()
w.CustomerName_Edit.wText = aCustomerName
w.Street_Edit.wText = aAddr
w.City_Edit.wText = aCity
w.State_Edit.wText = aRegion
w.Zip_Edit.wText = aPostalCode
w.OKButton.Click()
validateRecordEntry()
}
Don’t worry about trying to make this code actually work – it’s rather abstracted and
is for demonstration only, but your actual DDT code might be similar. What this small
amount of code does is very powerful and serves as a great example of a standard
DDT scenario.

As for the other three methods covered in the help (CallMethod, SetValue and
GetValue) I won’t talk about those now – they’re not necessary for this primer on
DDT.
The part of this whole process that’s not clearly covered is that of actually reading
the external data. That is in part because there are so many options. TestComplete
can read your DDT data from text files, spreadsheets, database tables or from
internal data arrays. To access your data, you can use ADO, OLE automation or
standard text file access methods. Many of these storage options and access
methods are shown in the DDT demo project included with TestComplete, which we’ll
cover shortly. For now, just plan on getting to your data whichever way is most
comfortable.

For more Software Testing Resources, please visit
Join largest Software Testing Community at
Over 5,000 Software Testing Interview Questions at
and
http://www.TestingInterviewQuestions.com
and
http://www.NewInterviewQuestions.com


Automation Framework Ebook

Diunggah oleh

Informasi Dokumen

Deskripsi Asli:

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Automation Framework Ebook

Diunggah oleh

Hak Cipta:

Format Tersedia

Software Testing : Automation Framework’s eBook from www.OneStopTesting.

Software Testing Automation

World’s Largest Portal on Software Testing Information & Jobs -

World’s Largest Portal on Software Testing Information & Jobs -

• Keywords, which are typically base level and describe generalized UI

Keyword driven testing is a very powerful tool helping organizations to do more

World’s Largest Portal on Software Testing Information & Jobs -

• Test development and automation must be fully separated – It is very

World’s Largest Portal on Software Testing Information & Jobs -

Building a collection of references that uniquely identify objects, sometimes

World’s Largest Portal on Software Testing Information & Jobs -

Vision for Automation

• Having a good methodology – It is important to have a good integrated

World’s Largest Portal on Software Testing Information & Jobs -

Success Factors for Automation

• Key Principles of Test Design

Comprehensive Automation Architecture

• Scope, assumptions, risks

• Test management who is responsible for managing the test process.

How to Measure Success

World’s Largest Portal on Software Testing Information & Jobs -

Some of the key quality metrics include:

• Coverage of system and requirements

World’s Largest Portal on Software Testing Information & Jobs -

World’s Largest Portal on Software Testing Information & Jobs -

1. Retrieving input data from storage.

Retrieving input data from storage

Entering data in an application form

World’s Largest Portal on Software Testing Information & Jobs -

World’s Largest Portal on Software Testing Information & Jobs -

Data Driven Automation Frameworks

Data Driven Scripts

Variable Data, Hard Coded Component Identification

Figure 1 is an example of activating a server-side image map link in a web

World’s Largest Portal on Software Testing Information & Jobs -

Highly Technical or Duplicate Test Designs

World’s Largest Portal on Software Testing Information & Jobs -

TestComplete's Support for Data-Driven Testing

GetCSVCount and GetCSVItem work with comma-separated strings in order to

02,c,”John”,”Q”,”Public”,13,”Product 02”,”1234 Anywhere St.”,”San Antonio”,”TX”

Here’s an important thing to keep in mind: GetCSVItem is what’s known as “zero-

varCustFirstName = GetCSVItem(‘02,c,”John”,”Q”,”Public”,13,”Product 02”,”1234

World’s Largest Portal on Software Testing Information & Jobs -

EnterCustomerRecord(lsCustomerName, lsAddr, lsCity, lsRegion, lsPostalCode)

World’s Largest Portal on Software Testing Information & Jobs -

World’s Largest Portal on Software Testing Information & Jobs -

For more Software Testing Resources, please visit

Join largest Software Testing Community at

Over 5,000 Software Testing Interview Questions at

World’s Largest Portal on Software Testing Information & Jobs -

Anda mungkin juga menyukai