Reliability. Can you trust the product in many and difficult situations?
Stability: the product s houldnt caus e cras hes , unhandled exceptions or s cript errors . Robus tnes s : the product handles fores een and unfores een errors gracefully. Stres s handling: how does the s ys tem cope when exceeding various limits ? Recoverability: it is pos s ible to recover and continue us ing the product after a fatal error. Data Integrity: all types of data remain intact throughout the product. Safety: the product will not be part of damaging people or pos s es s ions . Dis as ter Recovery: what if s omething really, really bad happens ? Trus tworthines s : is the products behavior cons is tent, predictable, and trus tworthy?
Attractivenes s : are all types of as pects of the product appealing to eyes and other s ens es ? Curios ity: will us ers get interes ted and try out what they can do with the product? Entrancement: do us ers get hooked, have fun, in a flow, and fully engaged when us ing the product? Hype: s hould the product us e the lates t and greates t technologies /ideas ? Expectancy: the product exceeds expectations and meets the needs you didn't know you had. Attitude: do the product and its information have the right attitude and s peak to you with the right language and s tyle? Directnes s : are (firs t) impres s ions impres s ive? Story: are there compelling s tories about the products inception, cons truction or us age?
Compatibility. How well does the product interact with software and environments?
Hardware Compatibility: the product can be us ed with applicable configurations of hardware components . Operating Sys tem Compatibility: the product can run on intended operating s ys tem vers ions , and follows typical behavior. Application Compatibility: the product, and its data, works with other applications cus tomers are likely to us e. Configuration Compatibility: products ability to blend in with configurations of the environment. Backward Compatibility: can the product do everything the las t vers ion could? Forward Compatibility: will the product be able to us e artifacts or interfaces of future vers ions ? Sus tainability: effects on the environment, e.g. energy efficiency, s witch-offs , power-s aving modes , telecommuting. Standards Conformance: the product conforms to applicable s tandards , regulations , laws or ethics .
Numbers
0 32768 (2^15) 32769 (2^15 + 1) 65536 (2^16) 65537 (2^16 +1) 2147483648 (2^31) 2147483649 (2^31 + 1) 4294967296 (2^32) 4294967297 (2^32 + 1) Scientific Notation (1E-16) Negative Floating Point/Decimal (0.0001) With Commas (1,234,567) European Style (1.234.567,89) All the Above in Calculations
Strings
Long (255, 256, 257, 1000, 1024, 2000, 2048 or more characters ) Accented Chars (, etc.) As ian Chars ( ) Common Delimiters and Special Characters ( ` | / \ , ; : & < > ^ ? Tab ) Leave Blank Single Space Multiple Spaces Leading Spaces End-of-Line Characters (^M)
SQL Injection ( s elect from cus tomer ) With All Actions (Entering, Searching, Updating, etc.) Right to Left Languages (Hebrew, Arabic)
General
Violates Domain-Specific Rules (an ip addres s of 999.999.999.999, an email addres s with no @, an age of -1) Violates Uniquenes s Cons traint
Web Tests
Navigation
Back (watch for Expired mes s ages and double-pos ted trans actions ) Refres h Bookmark the URL Select Bookmark when Logged Out Hack the URL (change/remove parameters ; s ee als o Data Type Attacks ) Multiple Brows er Ins tances Open
Input
See also Data Type Attacks HTML/JavaScript Injection (allowing the us er to enter arbitrary HTML tags and JavaScript commands can lead to s ecurity vulnerabilities ) Check Max Length Defined on Text Inputs > 5000 Chars in TextAreas
Syntax
HTML Syntax Checker (http://validator.w3.org/) CSS Syntax Checker (http://jigs aw.w3.org/cs s -validator/)
Preferences
Javas cript Off Cookies Off Security High Res ize Preferences Brows er Window Change Font Size
Testing Wisdom
A tes t is an experiment des igned to reveal information or ans wer a s pecific ques tion about the s oftware or s ys tem. Stakeholders have ques tions ; tes ters have ans wers . Dont confus e s peed with progres s . Take a contrary approach. Obs ervation is exploratory. The narrower the view, the wider the ignorance. Big bugs are often found by coincidence. Bugs clus ter. Vary s equences , configurations , and data to increas e the probability that, if there is a problem, tes ting will find it. Its all about the variables .
Heuristics
Variable Analysis - Identify anything whos e value can change. Variables can be obvious , s ubtle, or hidden. Touch Points - Identify any public or private interface that provides vis ibility or control. Provides places to provoke, monitor, and verify the s ys tem. Boundaries - Approaching the Boundary (almos t too big, almos t too s mall), At the Boundary Goldilocks - Too Big, Too Small, Jus t Right CRUD - Create, Read, Update, Delete Follow the Data - Perform a s equence of actions involving data, verifying the data integrity at each s tep. (Example: Enter Search Report Export Import Update View) Configurations - Varying the variables related to configuration (Screen Res olution; Network Speed, Latency, Signal Strength; Memory; Dis k Availability; Count heuris tic applied to any peripheral s uch as 0, 1, Many Monitors , Mice, or Printers ) Interruptions - Log Off, Shut Down, Reboot, Kill Proces s , Dis connect, Hibernate, Timeout, Cancel Starvation - CPU, Memory, Network, or Dis k at maximum capacity Position - Beginning, Middle, End (Edit at the beginning of the line, middle of the line, end of the line) Selection - Some, None, All (Some permis s ions , No permis s ions , All permis s ions ) Count - 0, 1, Many (0 trans actions , 1 trans actions , Many s imultaneous trans actions ) Multi-User - Simultaneous create, update, delete from two accounts or s ame account logged in twice. Flood - Multiple s imultaneous trans actions or reques ts flooding the queue. Dependencies - Identify has a relations hips (a Cus tomer has an Invoice; an Invoice has multiple Line Items ). Apply CRUD, Count, Pos ition, and/or Selection heuris tics (Cus tomer has 0, 1, many Invoices ; Invoice has 0, 1, many Line Items ; Delete las t Line Item then Read; Update firs t Line Item; Some, None, All Line Items are taxable; Delete Cus tomer with 0, 1, Many Invoices ) Constraints - Violate cons traints (leave required fields blank, enter invalid combinations in dependent fields , enter duplicate IDs or names ). Apply with the Input Method heuris tic. Input Method - Typing, Copy/Pas te, Import, Drag/Drop, Various Interfaces (GUI v. API) Sequences - Vary Order of Operations , Undo/Redo, Revers e, Combine, Invert, Simultaneous Sorting - Alpha v. Numeric, Acros s Multiple Pages State Analysis - Identify s tates and events /trans itions , then repres ent them in a picture or table. Works with the Sequences and Interruption heuris tics . Map Making - Identify a bas e or home s tate. Pick a direction and take one s tep. Return to bas e. Repeat. Users & Scenarios - Us e Cas es , Soap Operas , Pers onae, Extreme Pers onalities
Frameworks
Judgment Inconsistencies
Abs ences , and Extras with res pect to Internal External Specific, or External Cultural reference points . (James Lynds ay, Workroom Productions )
Observations
Input Output Linkage (James Lynds ay, Workroom Productions )
Flow
Input Proces s ing Output
Requirements
Us ers Function Attributes Cons traints (Gaus e & Weinberg Exploring Requirements )
Demings
Cycle Plan Do Check Act 'This cheat sheet includes ideas from Elisabeth Hendrickson, James Lyndsay, and Dale Emery' 'www.qualitytree.com Copyright 2006 Quality Tree Software, Inc.' _______________________________________________________________________________________________________
Function Testing
Tes t what it can do 1. 2. 3. 4. Identify things that the product can do (functions and s ubfunctions ). Determine how youd know if a function was capable of working. Tes t each function, one at a time. See that each function does what its s uppos ed to do and not whatit is nt s uppos ed to do.
Domain Testing
Divide and conquer the data 1. 2. 3. 4. Look for any data proces s ed by the product. Look at outputs as well as inputs . Decide which particular data to tes t with. Cons ider things like boundary values , typical values , convenient values , invalid values , or bes t repres entatives . Cons ider combinations of data worth tes ting together.
Stress Testing
Overwhelm the product 1. Look for s ub-s ys tems and functions that are vulnerable to being overloaded or broken in the pres ence of challenging data or cons trained res ources . 2. Identify data and res ources related to thos e s ub-s ys tems and functions . 3. Select or generate challenging data, or res ource cons traint conditions to tes t with: e.g., large or complex data s tructures , 4. high loads , long tes t runs , many tes t cas es , low memory conditions .
Flow Testing
Do one thing after another 1. Define tes t procedures or high level cas es that incorporate multiple activities connected end-to-end. 2. Dont res et the s ys tem between tes ts . 3. Vary timing and s equencing, and try parallel threads .
Scenario Testing
Tes t to a compelling s tory 1. Begin by thinking about everything going on around the product. 2. Des ign tes ts that involve meaningful and complex interactions with the product. 3. A good s cenario tes t is a compelling s tory of how s omeone who matters might do s omething that matters with the product.
Claims Testing
Verify every claim 1. 2. 3. 4. Identify reference materials that include claims about the product(implicit or explicit). Analyze individual claims , and clarify vague claims . Verify that each claim about the product is true. If youre tes ting from an explicit s pecification, expect it and the product to be brought into alignment.
User Testing
Involve the us ers 1. 2. 3. 4. 5. Identify categories and roles of us ers . Determine what each category of us er will do (us e cas es ), how they will do it, and what they value. Get real us er data, or bring real us ers in to tes t. Otherwis e, s ys tematically s imulate a us er (be carefulits eas y to think youre like a us er even when youre not). Powerful us er tes ting is that which involves a variety of us ers and us er roles , not jus t one.
Risk Testing
Imagine a problem, then look for it. 1. 2. 3. 4. 5. What kinds of problems could the product have? Which kinds matter mos t? Focus on thos e. How would you detect them if they were there? Make a lis t of interes ting problems and des ign tes ts s pecifically to reveal them. It may help to cons ult experts , des ign documentation, pas t bug reports , or apply ris k heuris tics .
Automatic Testing
Run a million different tes ts
1. Look for opportunities to automatically generate a lot of tes ts . 2. Develop an automated, high s peed evaluation mechanis m. 3. Write a program to generate, execute, and evaluate the tes ts
Scenario Testing
Tes t to a compelling s tory 1. Begin by thinking about everything going on around the product. 2. Des ign tes ts that involve meaningful and complex interactions with the product. 3. A good s cenario tes t is a compelling s tory of how s omeone who matters might do s omething that matters with the product.
Project Environment
''Creating and executing tests is the heart of the test project. However, there are many factors in the project environment that are critical to your decision about what particular tests to create. In each category, below, consider how that factor may help or hinder your test design process. Try to exploit every resource.''
Information. Information about the product or project that is needed for testing.
Are there any engineering documents available? Us er manuals ? Web-bas ed materials ? Does this product have a his tory? Old problems that were fixed or deferred? Pattern of cus tomer complaints ? Do you need to familiarize yours elf with the product more, before you will know how to tes t it? Is your information current? How are you appris ed of new or changing information? Is there any complex or challenging part of the product about which there s eems s trangely little information?
Matrices & Checklis ts : Are any documents needed to track or record the progres s of tes ting?
Product Elements
''Ultimately a product is an experience or solution provided to a customer. Products have many dimensions. So, to test well, we must examine those dimensions. Each category, listed below, represents an important and unique aspect of a product. Testers who focus on only a few of these are likely to miss important bugs.''
Platform. Everything on which the product depends (and that is outside your project).
External Hardware: hardware components and configurations that are not part of the s hipping product, but are required (or optional) in order for the product to work: CPU's , memory, keyboards , peripheral boards , etc. External Software: s oftware components and configurations that are not a part of the s hipping product, but are required (or optional) in order for the product to work: operating s ys tems , concurrently executing applications , drivers , fonts , etc. Internal Components : libraries and other components that are embedded in your product but are produced outs ide your project. Since you dont control them, you mus t determine what to do in cas e they fail.
Operational Criteria
Capability: Can it perform the required functions ? Reliability: Will it work well and res is t failure in all required s ituations ? Error handling: the product res is ts failure in the cas e of errors , is graceful when it fails , and recovers readily. Data Integrity: the data in the s ys tem is protected from los s or corruption. Safety: the product will not fail in s uch a way as to harm life or property. Usability: How eas y is it for a real us er to us e the product?
Learnability: the operation of the product can be rapidly mas tered by the intended us er. Operability: the product can be operated with minimum effort and fus s . Accessibility: the product meets relevant acces s ibility s tandards and works with O/S acces s ibility features . Security: How well is the product protected agains t unauthorized us e or intrus ion? Authentication: the ways in which the s ys tem verifies that a us er is who s he s ays s he is . Authorization: the rights that are granted to authenticated us ers at varying privilege levels . Privacy: the ways in which cus tomer or employee data is protected from unauthorized people. Security holes: the ways in which the s ys tem cannot enforce s ecurity (e.g. s ocial engineering vulnerabilities ) Scalability: How well does the deployment of the product s cale up or down? Performance: How s peedy and res pons ive is it? Installability: How eas ily can it be ins talled onto its target platform(s )? System requirements: Does the product recognize if s ome neces s ary component is mis s ing or ins ufficient? Configuration: What parts of the s ys tem are affected by ins tallation? Where are files and res ources s tored? Uninstallation: When the product is unins talled, is it removed cleanly? Upgrades: Can new modules or vers ions be added eas ily? Do they res pect the exis ting configuration? Compatibility: How well does it work with external components & configurations ? Application Compatibility: the product works in conjunction with other s oftware products . Operating System Compatibility: the product works with a particular operating s ys tem. Hardware Compatibility: the product works with particular hardware components and configurations . Backward Compatibility: the products works with earlier vers ions of its elf. Resource Usage: the product does nt unneces s arily hog memory, s torage, or other s ys tem res ources .
Development Criteria
Supportability: How economical will it be to provide s upport to us ers of the product? Testability: How effectively can the product be tes ted? Maintainability: How economical is it to build, fix or enhance the product? Portability: How economical will it be to port or reus e the technology els ewhere? Localizability: How economical will it be to adapt the product for other places ? Regulations: Are there different regulatory or reporting requirements over s tate or national borders ? Language: Can the product adapt eas ily to longer mes s ages , right-to-left, or ideogrammatic s cript? Money: Mus t the product be able to s upport multiple currencies ? Currency exchange? Social or cultural differences: Might the cus tomer find cultural references confus ing or ins ulting? ''Designed by James Bach, james@satisfice.com, www.satisfice.com, Copyright 1996-2006, Satisfice, Inc.'' _______________________________________________________________________________________________________
Interopeability tour: What does this application interact with? Data tour: Identify the major data elements of the application. Structure tour: Find everything you can about what compris es the phys ical product (code, interfaces , hardware, files , etc...). Based upon an idea by Michael Kelly http://www.michaeldkelly.com _______________________________________________________________________________________________________
HICCUPPS
Consistency (this agrees with that) - an important theme in oracles
History: If a product is incons is tent with previous vers ions of its elf, we s us pect that there might be a problem. Image: If a product is incons is tent with an image that the company wants to project, we s us pect a problem. Comparable Products: When a product s eems incons is tent with a comparable product, we s us pect that there might be a problem. Claims: When a product is incons is tent with claims that important people make about it, we s us pect a problem. User Expectations: When a product is incons is tent with expectations that a reas onable us er might have, we s us pect a problem. Purpose: When a product is incons is tent with its des igners explicit or implicit purpos es , we s us pect a problem. Product: When a product is incons is tent internally as when it contradicts its elf we s us pect a problem. Statutes and Standards: When a product is incons is tent with laws or widelyaccepted or relevant s tandards , we s us pect a problem. Orignal idea from James Bach and Michael Bolton http://www.develops ens e.com/articles /2005-01-Tes tingWithoutAMap.pdf