Skill Category 8

Testing Software
Developed by Contractors
Definitions

•COTS = Commercial off the shelf software.

•Contractors = Outsourcers or offshore software developers that produce

COTS software.

•Contracting Organization = The customer, group or organization that

purchases the COTS software.
Challenges in Testing Acquired Software

There is a trend in the software industry for organizations to move from in

house developed software to COTS software and software developed by
contractors.

Differences in software developed in-house and by

contractors

•Relinquishment of control

•Loss of control over reallocation of resources

Challenges of Testing COTS Software

•There may be tasks or items missing

•The software fails to perform

•The software has extra undocumented features

•The software does not meet business needs

•The software does not meet operational needs

•The software does not meet customer needs

Differences of Testing in-house Software vs. Software
Developed by a Contractor

•Quality factors may not be specified

•Software has not testable requirements or criteria

•Contractors organization standards may not be met

•Software has missing requirements

•Overlooked changes in standards of technology

•Inadequate training and deployment methods from contractor

Differences in Offshore Contract Organizations

50% of the software developed by offshore organizations fails to meet the

expectation of the contracting organization.

Offshore Testing Differences

•Cultures are not the same

•Communication Barriers

•Loss of Employee Morale and Support

•Root Causes of Contractor IT Organization are not Addressed properly

Software Testers Responsibility for Software developed
by a Contractor
•Even though the software is being developed by a contractor, the quality
of the product can’t be contracted, that is the responsibility of the
software tester.

•The tester should accept the responsibility for software testing

developed by the contractor.

•The tester’s responsibility should include assuring the process of

selecting COTS software and working with the contractor are adequate.

•The tester needs to examine the contractor tests in relation to the SEI
CMMI Capability Maturity Model.

Level 1 Maturity=Greater risk of defects

Level 5 Maturity=More stable product
COTS Software Test Process
The amount of testing performed on this type of software will depend on the risk
of placing this software into production.

Process to Test High-Risk COTS Software (consists of)

•Assure completeness of needs specification

•Define critical success factor

•Determine compatibility w/current computer environment

•Assure software can be integrated into your business system work flow

•Demonstrate the software in operation

•Evaluate the people fit

•Acceptance test the COTS software

Assure Completeness of Needs Specification (COTS Software
Test Process)

This step determines whether you have adequately defined your needs. Needs are
defined by the categories below.

Output products and reports –

Output products and reports are specific documents that you want produced by the
computer system.

Management decision information –

This category tries to define the information needed for decision-making purposes.
Define the Critical Success Factors (COTS Software Test Process)

This step determines whether the software package will be successful in meeting
your needs.

Examples of common CSF’s

•Ease of use

•Expandability

•Maintainability

•Cost Effectiveness

•Transferability

•Reliability

•Security
Determine Compatibility with your Computer Environment
(COTS Software Test Process)

Best performed by preparing a checklist defining your compatibility needs.

Hardware Compatibility –

• Hardware Vendors
• Type of Printers
• Keyboard Restrictions

Operating Systems Compatibility –

• Name of OS
• Version of OS

Data Compatibility –
In many cases software compatibility will answer the questions of data compatibility
Assure Software can be integrated into the Businesses
System Work Flow (COTS Software Test Process)

•The objective of this process is to illustrate the type and frequency of work flow
changes that will be occurring.

•At the end of testing you will need to make a determination of whether you are
pleased with the revised work flow. If you are not pleased the testing may need to
continue. If so you should prepare a clean data flow diagram that indicates what
actions need to take place to integrate the COTS system into the organizations
work flow.
Demonstrate the Software in Operation (COTS Software Test
Process)

This step analyzes the many facets of Software.

Demonstrations of the software can be done in two ways -

•Computer store controlled demonstration

•Contractors organization site demonstration

Aspects of computer software observed during the demonstration –

•Understandability
•Clarity of communication
•Ease of use of instruction manual
•Functionality of the software
•Knowledge to execute
•Effectiveness of help routines
•Evaluate program compatibility
•Data compatibility
•Smell Test
Evaluate the People Fit (COTS Software Test Process)
The objective of this step is to determine whether you employees can use the
software.

Results of this test will show the following –

•The software can be uses as is

•Additional training and support is necessary

•The software is not usable with the skill sets of the proposed users

Acceptance Test the COTS Software (COTS Software Test

Process)

There is little difference between acceptance testing in-house developed software

and acceptance testing acquired software.
Contracted Software Test Processes
Good test practices for software development involve acquiring organization’s
testers in all phases of development and operation.

The following tester responsibilities are identified for best practices –

•Assure the process for contracting software is adequate

•Assure the requirements and contract criteria are testable
•Review the contractors test plan
•Perform acceptance testing on the software
•Insure the software meets the needs of the organization it is targeted for
•Ensure knowledge transfer occurs and intellectual property is protected
•Incorporate copyrighted materials into the contracting organizations manuals
•Assure the ongoing operation and maintenance of the contracted software
•Assure the effectiveness of contractual relations
Assure the Process for Contracting Software is
Adequate (Contracted Software Test Processes)

Contracts for software testing are legal documents and should

include the following information –

•What is done

•Who does it

•When it is done

•How it is done

•Where it is done

•What are the penalties for nonperformance

Concerns that should be addressed by a contracting process will include the
following factors –

•Warranty- Guarantee provided by contractor that deliverables meet the specifications

•Deliverables- These should be described in great detail
•Delivery date- Date on which the software is delivered (duh0
•Commencement Date- Date that the contract becomes effective
•Installation- Commitment to install the tested software
•Updates- Any planned or unplanned updates need to be addressed
•Contractor Support-Types, quantity and location need to be specified
•Costs-Amount that is agreed upon to be paid to the contractor
•Foreign Attachment-Interconnections with other systems or contracts should be addressed
•Penalties-Cost of failures from either party in the contract need to be spelled out
•Life of Contract-Duration of the contract
•Modification of Capability-Contracts needs to clarify types of modifications contractor is willing
to make to the system
•Service Discontinuance-Termination of contractor services needs to be specified
•Manual/Training Discontinuance-End date for training and documentation
•Acceptance Test Criteria-Criteria that determines that the delilvered product is acceptable
•Purchase vs. Lease-Will software be bought or leased
•Fairness of Contract-Is the contract fair
•Performance of Maintenance-Who and when will be maintenance on the software performed
•Contractor Training-Type, frequency and caliber
•Contractor Manuals-Documentation needed to operate the system
•Supplies-What is needed to keep the system operational
Concerns that should be addressed by a contracting process will
include the following factors – (continued)
•Transportability-Moving the system from one location to another
•Terminations-Ending the contract
•Contractor Employee Rights-People needed to fullfill contract specifications
•Governing Law-State in which the contract takes place
•Contractor Inspection-Right of the contractor to inspect the records of the contracting
agency
•Access to Developers and Documentation
•Assure Requirements and Criteria of Contract is Testable
Review the Adequacy of the Contractors Test Plan
(Contracted Software Test Processes)

This Responsibility is to evaluate the adequacy and completeness of testing that

will be performed.

The software tester should develop an opinion on the contractor’s ability to deliver
software that meets the requirements or criteria in the contract.

Assure Development is Effective and Efficient (Contracted

Software Test Processes)

Perform Acceptance Testing on the Software (Contracted

Software Test Processes)

The software should not be allowed to be put into place without acceptance testing
done by the software tester.

The extent of acceptance testing will be dependent upon the risk associated with
the use of the software being contracted.
Issue a Report on the Adequacy of the Software to Meet
the Needs of the Organization (Contracted Software Test
Processes)

This report should focus on the 2 definitions of quality –

•The software adequately meets the requirements it is being designed for.

•The software is fit for use by the intended users

Ensure Knowledge Transfer & Intellectual Property

Rights are Protected (Contracted Software Test Processes)

Two concerns software tester must address concerning this –

•There is adequate knowledge transfer about the software from the developer to
the customer.

•The intellectual property rights of both the contractor and contracting organization
are protected.
Assuring the Ongoing Operation & Maintenance of the
Contracted Software (Contracted Software Test Processes)

The major concern during the operation and maintenance of a purchased or leased
application system is the both parties to the agreement comply w/ the contractual
requirements.

The Major areas of concern durning the maintenance period as

follows –
•Adequacy of control

•Adequacy of documentation

•Speed of service

•Nearness of service

•Competency of service

•Adequacy of hardware

•Skilled personnel
Assuring the Ongoing Operation & Maintenance of the
Contracted Software (continued)
•Multi-contractor problem resolution

•Cost of services performed

•Cost of operations

•Error diagnosis

•Error documentation
Assure the Effectiveness of Contractual Relations
(Contracted Software Test Processes)

The relationship between the contractor and the contracting organization is an

ongoing relationship.

Concerns that foster a relationship of harmony and good that

testers can evaluate are – (you have got to be kidding)

• Have the contractor obligations met

• Have the contracting organizations obligations met
• Have the software needs been met
• Are there Limits on cost increases
• Are there options for added features or work
• Are there provisions for renegotiation
• Is there compensation for errors caused by the contractor
• Contractual termination procedures should be followed according to contract
requirements