Continuous Integration

WITH FITNESSE AND SELENIUM

By Brian Kitchener briank@ecollege.com

Intro
Who am I? Overview Continuous Integration The Tools Selenium Overview Fitnesse Overview Data Dependence My Solution Demo

What is CI?
From Wikipedia:

Continuous Integration (CI) implements a continuous processes of applying quality control - small pieces of effort, applied frequently. Continuous integration aims to improve the quality of software, and to reduce the time taken to deliver it, by replacing the traditional practice of applying quality control after completing all development.
automatic code integration and validation! A Build Server to control everything and store reporting A development/deployment process that supports being automated. Scripts to do the work One or more test frameworks to test the code (nUnit, Fitnesse, Selenium) A minimal set of tests that cover a broad scope of functionality Automatic testing and validation for each code change Immediate and automatic isolation of the cause Up-to-date, centralized reporting You know who broke it, and who needs to fix it. You never will test a broken environment again.

In English:

What do we need?

What do we get?

The Tools
Build Server (Hudson) Runs our Tests Unit Tests

Tests the individual functions

Fitnesse Tests:
Database (DB Fit) API calls Web Services

Selenium Tests:
UI Issues Functionality problems Javascript Errors

Firefox only plugin, no other browsers. Step based testing, no complex functionality

Record and Playback

Fast test prototyping

Troubleshooting
Can be run from Hudson using a plug-in.

Selenium IDE

Stored on your websites web server Executable via a url Supports any browser Tests stored in a central repository Has problems supporting https or changing domains

Selenium Core

Selenium RC
Client/server java application needs to be installed Supports any Browser/OS combination Test can be run locally or remotely Controlled and executed through xUnit Test written in your favorite programming language/IDE (C#, java, python, ruby) Access to full features of that language and any extra libraries

[Test] public void testGoogle() { selenium.Open(http://www.google.com/"); selenium.waitForPageToLoad(30000); selenium.Type("q", "selenium"); selenium.Click("btnG"); selenium.waitForPageToLoad(30000); Assert.IsTrue(selenium.IsTextPresent("Selenium web application testing system")); }

Distributed parallel testing More setup, but faster execution Run on multiple machines simultaneously Generates more load

Selenium Grid

How Selenium Works

Pros

Cons

Injects itself as javascript into the page object

Injects itself as javascript into the page object

Any browser that supports javascript supports selenium

The ONLY tool that can work with any browser/os Direct access to page objects

Non Firefox browsers dont always work perfectly It cant see anything outside of the page

Extremely fast execution time

Cant click on the menu, navigation buttons, or any extra windows Predefined routines reproduce this functionality Changing domains/https

Doesnt hijack your computer

You can do other things while its running

XSS Limitations

Full access to outside libraries

It fakes a lot

If you need something else, you can code it.

It doesnt use the mouse/keyboard

Pick your language Record in IDE and convert to RC.

Easier for non-coders Fast test prototyping Troubleshooting

Doesnt really click on anything Accesses each item directly and calls a function Can bypass front end functionality such as onclick

Can access hidden items even if you dont want it to But can be gotten around

Open source tools can be customized

Some functionality cant be automated

Open source tools require a lot of coding

Fitnesse Overview
What is Fitnesse? Fitnesse = FIT + Wiki
FIT : Framework for Integrated Testing - The engine that translates tables into actions Wiki: The GUI that allows the user to edit and run tests

A website that allows us to create, edit, and run tests. The tests must be written as a table

Each row is a test

Fitnesse continued

Typically used to test an application or database directly, without accessing the GUI.
Calls a Web service, API, DB Procedure Test the Business Logic directly. We dont have to try to trick the ui into an incorrect state Typically more stable than GUI testing as less functionality changes over time

Requires the creation of a Fixture which knows how to translate the test into an action.
The fixtures are many times written by the application developers Some standard fixtures exist, such as the RESTFixture for testing REST based services Once written, the fixtures can be largely ignored Different fixtures allow different functionality.

Fitnesse continued

Typical data driven testing - same test multiple times for all data types
Boundary conditions Error code checking Business Logic

Acceptance TestingWiki style input allows non-coders to develop and run tests. Business and Product personnel can (theoretically) write acceptance tests However it requires a knowledge of the AUT

GUI testing is possible with appropriate fixture

Cant test for javascript errors without a GUI test However selenium is more robust

Fitnesse

Fitnesse pros and cons

Tests are editable and executable from a web page. Uses Xpath. Can generate fake bugs (stuff looks broken because of a Fitnesse

problem) Test maintenance can be time consuming

Everything is a text document Same tests many different places

Global variables allow you to change url, user, etc very easily Many tests can require unique information (id, name) that can

change in different environments. Run-time variables allow you to create more dynamic tests. Knowing what data should be returned can be troublesome Can generate exact error-causing conditions Useless if your application sucks

Data Dependence
The biggest problem with automation is changing data

Time changes, items expire, dates change. Need to test in different environments going to out to production. Many items are referenced by an ID that is unique, or increment. Need to know what are the expected results of your test. If something you assume is incorrect, your tests will fail.

User not there Items expired Dates changed GUI item changed or vanished

In unit testing, you can mock data to test with. When testing a real system, your data can fluctuate

DB Refresh Virtualized environments User activity can change your data set Date oriented things expire Ids increment Using external or live data

Data Options
Set our data manually each run. DB Copy/refresh to return to a known state. Start from and return to a known state CRUD only your data Get current data Connect to the db to grab data. Run UI automation to get data. Generate data Use automation to generate a data set at run time

My Solution
Data Generation Suite Can point to any client/environment. Checks to see if my data is there, if it is it deletes it. Then creates all data fresh, and sets state. A Smoke Test to be run by the Hudson server. Checks all major functionality. Checks each of the functional areas Doesnt validate what data it gets. (it fluctuates too much) A Regression Test that tests full functionality. Data Generation Suite needs to be run first Tests full functionality and validates data is correct. Fitnesse tests that check the services Use the data selenium created. Verify boundary conditions Verify response code and format. Verify errors

The end result

All builds automatically are smoke tested for viability. Takes ~15 minutes to regression test a build in each

environment Can change users, clients, environments, configuration. All tests dont always pass in all environments, have to check test results. Can use Test Driven Development methodology

Create tests first Code is good when they pass

Relying heavily on automation Fast turnaround. High confidence level for code changes, new services

In progress
Work on new framework Migrate to Selenium2 (mobile browser support) Fully implement PageObjects Use a VS config file Support multiple browsers