Auditing Software Testing Process In this tutorial you will learn about Auditing Software Testing Process - Introduction,

Types of Testing Process Audits, Audit to verify compliance, Audit for process improvement/problem solving, Audit for Root Cause Analysis, Internal Audits, External Audits, and Why Audit Software Testing Process? How to Audit, What can be audited and Summary. Introduction: To ensure transparency and reliability of the IT systems it may be necessary to audit the Software Development Processes including the most important aspect Software Testing Process. Auditing is an important activity in organizations. In the context of testing it helps us ensure that the Testing processes are followed as defined. Types of Testing Process Audits There can be various reasons to conduct Audits. The Audits may serve aim to achieve certain definite goals. Based on that we can classify them as follows: Audit to verify compliance: In this type of auditing the prime motivation is to judge if the process complies with a standards. In these scenarios, the actual testing process is compared with the documented process. For example ISO Standards require us to define our Software testing process. The audit will try to verify if we actually conducted the testing as documented Audit for process improvement/problem solving: In this type of audit the motivation is to audit and trace the various steps in the process and try to weed out process problems. For instance it is observed that too many software defects escaped detection even though the testing process was apparently followed. So the audit is done as a preliminary step to collect facts and analyze them. Audit for Root Cause Analysis In this type of audit the motivation is to audit the testing process is to find a Root Cause of a specific problem. For example the customers discovered a huge problem with the software. So we retrace our testing steps to find out what went wrong in this specific case. Internal Audits Typically the internal audits are initiated from within the organizations External Audits External Audits are done by and initiated by external agencies Why Audit Software Testing Process?

Auditing Test Process helps the management understand if the process is being followed as specified. Typically Testing audit may be done for one or more of the following factors: To ensure continued reliability and integrity of the process To verify compliance of standards (ISO, CMM, etc) To solve process related problems To find the root cause of a specific problem To detect or prevent Fraud To improve the Testing process Auditing of the Testing process may also be done if the Software Product is a mission critical one such as used for Medical Life Support Systems This is done to prevent any loop holes or bugs in the system How to Audit Typically the Audit of the Testing Process will include the following steps: reviewing the Testing process as documented in the Quality Manual This helps the auditor understand the process as defined. Reviewing the deliverable documents at each step Document reviewed include ............... Test Strategy ............... Test Plans ............... Test Cases ............... Test Logs ............... Defects Tracked ............... Test Coverage Matrix ............... any other relevant records Each of the above document provides a certain level of traceability that the process was followed and the necessary steps were taken Interviewing the Project Team at various levels PM, Coordinator, Tester Interviewing the Project Team members gives an understanding of the thought process prevalent in those conducting the Testing Process. This can provide valuable insights over an above what was actually documented ISACA ww.isaca.org provides guidelines and standards for Auditing Information Systems & Software Development Lifecycle CISA stands for Certified Information Systems Auditor Similarly independent agencies may verify the Test Processes and SDLC for ensuring compliance with FDA ( Food and Drug Administration) What can be audited?

Whether the test process deliverables exist as specified The only thing that can be really verified in an audit is that the process deliverables exist. The process deliverables are taken as a proof that the necessary steps were taken to do the testing. For example if Test Logs exist, we assume that testing was done and the Test Logs were created as a result of actual tests executed. A separate exercise may be initiated to verify the authenticity of the Test Logs or other test deliverables whether test cases created covered all requirements/use cases This analysis reveals if the test coverage was sufficient. It indicates that whether the testing team did the best to provide adequate amount of testing whether all Defects were fixed The Status of all the Defects logged is checked to verify if all were fixed and verified Whether there are any known bugs in the software released Sometimes all the defects may not be fixed, the software may be released with known problems. Test Logs would indicate the actual results and evidence of any bugs being present. Whether the levels of testing was effective enough If Defects pass thru the various levels of testing undetected, it may reflect poorly on the effectiveness of the testing process

What were the number of defects (Defect Leaks) that went by undetected in each phase Number of iterations of testing in each level Time taken to test each module/component This data may be used for process improvement Versions of source code actually tested

The Test Logs and Defect Logs may indicate (if the information was captured) the actual versions of code/components tested. This information may be valuable in root cause analysis. Challenges in Testing Web Based Applications In this tutorial you will learn about Challenges in Testing Web Based Applications Introduction, Why testing Web Applications is different? Factors effecting Testing of Web Applications, Why technology platforms affect testing? Challenges in Testing Web Based Web Applications, Summary. Introduction:

Web based Applications are increasingly becoming more feature rich, important and also the most popular means for developing commercial systems. Most companies opt for developing web based software wherever possible. This helps in catering to large number of end users. The deployment of the apps (once the infrastructure is in place) is fairly easy. The web based applications are powerful and have the ability to provide feature rich content to a wide audience spread across the globe at an economical cost. Hence it is a daunting task to test these applications and with more and more features testing these apps is becoming even more complex. In this article we will study the challenges faced when testing these applications Why testing Web Applications is different? Testing web applications is different because of many factors scenarios affecting the performance and user experience. Web applications can typically cater to a large and a diverse audience. Web Applications can also be exposed to wide range of security threats. Web applications may open up illegal points of entry to the databases and other systems holding sensitive information. To ensure that the web application works reliably and correctly under different situations these factors need to be accounted for and tested. Hence a lot of effort needs to put in for Test Planning and Test Design. Test Cases should be written covering the different scenarios not only of functional usage but also technical considerations such as Network speeds, Screen Resolution, etc. For example an application may work fine on Broad Band internet users but may perform miserably for users with dial up internet connections. Web Applications are known to give errors on slow networks, whereas they perform well on high speed connections. Web pages dont render correctly for certain situations but work okay with others. Images may take longer to download for slower networks and the end user perception of the application may not be good.

Factors effecting Testing of Web Applications: As mentioned above Web Applications can have a lot of variables affecting them such as: - Numerous Application Usage (Entry Exit) Paths are possible Due to the design and nature of the web applications it is possible that different users follow different application usage paths.

For example in an online banking application a user may directly go to Bill Pay page and other users may check account balances, view previous transactions and then Pay the Bills. Generally a large number of usage paths are possible and all are supposed to work well. All these Permutations and Combinations need to be tested thoroughly - People with varying backgrounds & technical skills may use the application Not all applications are self explanatory to all people. People have varying backgrounds and may find the application hard to use. For instance a Business Intelligence application with Drill-Down-Reports may work out for certain users but not for others. Although this affects the design of the applications, but these factors should be tested in usability testing of the applications - Intranet versus Internet based Applications Intranet based applications generally cater to a controlled audience. The developers and architects can make accurate assumptions about the people accessing the apps and the hardware/software/technical specifications of the client machines. While it may be difficult to make similar assumptions for Internet Based Applications Also the intranet users can generally access the app from trusted sources whereas for internet applications the users may need to be authenticated and the security measures may have to be much more stringent. Test Cases need to be designed to test the various scenarios and risks involved. - The end users may use different types of browsers to access the app Typically for internet based applications users may have different Browsers when accessing the apps. This aspect also needs to be tested. If we test the app only on IE then we cannot ensure if works well on Netscape or Fire-Fox. Because these browsers may not only render pages differently but also have varying levels of support for client side scripting languages such as java-script. - Even on similar browsers application may be rendered differently based on the Screen resolution/Hardware/Software Configuration - Network speeds: Slow Network speeds may cause the various components of a Webpage to be downloaded with a time lag. This may cause errors to be thrown up. The testing process needs to consider this as important factor specially for Internet based Applications - ADA ( Americans with Disabilities Act) It may be required that the applications be compliant with ADA. Due certain disabilities, some of the users may have difficulty in accessing the Web Applications

unless the applications are ADA compliant. The Application may need to be tested for compliance and usability - Other Regulatory Compliance/Standards: Depending on the nature of the application and sensitivity of the data captured the applications may have to be tested for relevant Compliance Standards. This is more crucial for Web Based Applications because of their possible exposure to a wide audience. - Firewalls: As mentioned earlier Applications may behave differently across firewalls. Applications may have certain web services or may operate on different ports that may have been blocked. So the apps need to be tested for these aspects as well. - Security Aspects: If the Application captures certain personal or sensitive information, it may be crucial to test the security strength of the application. Sufficient care need to be taken that the security of the data is not compromised. Why technology platforms affect testing? Technology platform upon which the Web Application was built also creates different strengths and weaknesses. Different Test Automation tools & packages are available for different Technology Platforms. This can influence the Test Strategy and the way in which Test Execution is done. Challenges in Testing Web Based Web Applications: To ensure that sufficient Test Coverage is provided for Web Based Applications and to provide a secure, reliable application to the user the above factors need to be considered. For internet based applications accessibility and security can be important issues. On one hand Organizations would like to cater to their users around the world on the other hand they could end up exposing their security holes all over. Testing could be the last chance to ensure the safety of the data and the organization.

Software Testing Myths Any IT professional is sure to know the different phases of the Software Development Life Cycle or SDLC, namely Feasibility Study, Requirement Analysis, Design, Construction or Coding, Testing, Implementation, Maintenance & Support - the activities carried out in each phase & their significance. But, very few agree on the importance of Software Testing phase.
Software implementation is a cozy bonfire, warm, bright, a bustle of comforting concrete activity. But beyond the flames is an immense zone

of darkness. Testing is the exploration of this darkness. - extracted from the 1992 Software Maintenance Technology Reference Guide Testing is often considered as a thankless job. While developers say with pride: "Wow!! My code is running in production", testers usually dont say "Wow!! The code that I tested is running in production"!!! This attitude can also be justified if we consider some examples of the usual talk that goes on among colleagues/peers/friends in the IT circle, like: Mr. A: Which project are you working on? Mr. B (Tester): Currently. I'm in a Testing project. Mr. A: Oh...Umm...OK... Mr. A: Mr. C, how about you? Mr. C (Programmer/Developer): A Development & Maintenance project Mr. A: Oohh?? What technology? Which platform? Whats the project all about?? And so on Even though there's no denying the fact that Construction/Coding is a very significant phase in the life cycle of any software product, the role of Testing as an activity should be given its due importance, because of the main reason, among others, that its the phase of SDLC just before a software product goes to production; i.e.; when your software product/application actually starts functioning in real world. Therefore, its the only phase where you can ensure & gain a reasonable level of confidence that that you are delivering quality products to the customer. This does not mean that quality assurance starts in the Testing phase. Ensuring & Maintaining quality is a continuous process which should be practiced right from Day 1 of the SDLC cycle. In other words, during testing phase, you can "bang" your product in different ways, by which I refer to different kinds of testing like functional testing, stress testing, performance testing & so on, and validate whether the software product works fine. Needless to say, delivering quality software delights customers & helps in getting more business. That said about the importance of Testing, we will now proceed to look into some perceptions or myths about software testing which prevails, in general, among the IT community Testing is done to demonstrate that there are no errors/bugs/defects in the software product being developed No. Not at all!! Though this is one way of conveying the meaning, it takes us away from the main objective of testing Testing is the process of uncovering defects... The objective of any tester should be to try his best to crack the code. This should not be seen as a destructive activity which points fingers at or find faults with the developers. In fact, testing should be considered as a healthy feedback mechanism to the developer community so that

they can make maximum use of the defects found during testing, analyze them, find the root cause & devise appropriate preventive mechanisms. Actually, the defects found during testing helps improving the quality of the code! A tester should never work under the assumption that the system works!! The software product should be tested with the intent of finding errors. The test, which when executed reveals a defect in the software, is a success. Though Testing, by itself, does not improve the quality of the software product, it is an indicator of the quality. Rather that considering Testing as a separate phase in the SDLC, it should be an activity which is integrated into each & every phase. The intent should be to Analyze & Test, Design & Test, Develop & Test, Fix Defects & Test more...!! Testing is done to ensure that the software/application does what it is supposed to do True enough!! But its not just that...The software should also not do what it is not supposed to do. Good test cases are the only way to establish a reasonable level of confidence on the software product. The Tester should think of all possible scenarios based upon the test requirements or test plan. Though there are various methods to prepare a large number of test cases like Equivalence Partioning, Boundary Value Analysis and so on, a lot depends on one's own intuition to come up with some good cases which reveals defects in the software Testing is easy...!!! Sometime back, I conducted a short class on Software Testing to a group of new recruits. While we were discussing about different type of projects, one participant said that he is very interested in a testing career. When asked the reason for his interest, prompt came the answer: Testing is very easy, thats why!! This holds good only in some situations. Though its simple to prepare straightforward test cases, at times testing can be a real challenging task. Any production issues will, in many cases, backfire first to the testing teams. Why was this scenario not covered in the test plan?? Therefore, a Tester should develop the capability to look or think beyond the requirements mentioned in the test plan or specifications. This is very important in case of System Testers who are responsible for ensuring that the software product works appropriately from "end-to-end ". Testing does not offer any opportunities for career growth. There are a wide range of roles that one can take up, if opting for a Testing career. Pursuing a testing career offers more scope for improving Business/Domain knowledge. It enables one to adopt a holistic approach of the entire software system instead of

concentrating on just a unit or module. A good number of testing certifications are offered by reputed institutes, which helps you attain a strong foundation in this career path. This does not mean that you dont have all these opportunities at all if you are a Developer. It can be said that a Testing career has its own plus points. It entirely depends on one's own interests! Testers do not perform well in development projects (or Testers are poor in coding) Tester or not, the expertise on developing quality code depends upon one's own programming skills and constant or continuous learning in that area. Some people, though in testing projects, take time out of their work to improve their programming skills by contributing to coding efforts or taking up projects voluntarily. Thus, being a Tester does not prevent you in any way from being an expert Programmer or Vice Versa!!! One way to narrow the communication gap between Tester & Developer community is to include the Testing teams, right from the Requirements/Design Stage meetings so that everyone involved in the life cycle of developing a software product can take part in the discussion & offer valuable suggestions. This is evident from the below lines, which wonderfully describes the importance of Testing More than the act of testing, the act of designing tests is one of the best bug preventers known. The thinking that must be done to create a useful test can discover and eliminate bugs before they are coded - indeed, testdesign thinking can discover and eliminate bugs at every stage in the creation of software, from conception to specification, to design, coding and the rest. -Boris Beizer, Software Testing Techniques, "Creating a Software Engineering Culture" by Karl Eugene Wiegers" No formal training is needed to work in a Testing project... Anyone can be a Tester! True enough, anyone can be a Tester...but, only a good Tester can come up with quality Test cases (just like how an expert Developer can write quality Code). It is essential that proper training is imparted to everyone joining Testing projects. This would not only helps one to understand the importance of Testing, but also tune one's mind to the requirements of becoming a good Tester, which would greatly contribute to a good career in Testing Testing for Agile Software Development

In this tutorial you will learn about Testing for Agile Software Development Background, Understanding Agile Software Development, How is Testing approach different in an Agile Development Scenario? What to test? Typical bugs found when doing agile testing? Steps Taken to Effectively Test in Agile Development Methodology, Ensuring software test coverage and Summary Background: To understand the Testing Process in an Agile Development Methodology, it is important to understand the Agile Development paradigm. Agile Development paradigm is not very new. Although the Agile Software Development Manifesto came into existence in February 2001, the concepts existed long before that and were expressed in different ways. Spiral Development Methodology is one such example. Understanding Agile Software Development: The Agile Software Development primarily focuses on an iterative method of development and delivery. The developers and end users communicate closely and the software is built. A working piece of software is delivered in a short span of time and based on the feedback more features and capabilities are added. The focus is on satisfying the customer by delivering working software quickly with minimum features and then improvising on it based on the feedback. The customer is thus closely involved in the Software Design and Development Process. The delivery timelines are short and the new code is built on the previous one. Despite this, high quality of the product cannot be comprised. This creates a different set of challenges for Software Testing. How is Testing approach different in an Agile Development Scenario? The Testing Strategy and Approach in Agile Development could be very different from traditional bureaucratic methods. In fact it could vary with project needs and the project team. In many scenarios, it may make sense to not have a separate testing team. The above statement should be understood carefully. By not having a testing team we do not consider testing to be any less important. In fact testing can done more effectively in short turn around times, by people who know the system and its objectives very well. For example in certain teams Business Analysts could be doing a few rounds of testing each time the software version is released. Business Analysts understand the Business Requirements of the Software and test it to verify if it meets the requirements. Developers may test the software. They tend to understand the system better and can verify the test results in a better way. Testing for AGILE Software Development requires innovative thinking and the right mix of people should be chosen for doing the testing.

What to test? Given the relatively short turn around times in this methodology it is important that the team is clear on what needs to be tested. Even though close interaction and innovation is advocated rather than processes, sufficient emphasis is given to the testing effort. While each team may have its own group dynamics based on the context, each code has to be unit tested. The developers do the unit testing to ensure that the software unit is functioning correctly. Since the development itself is iterative it is possible that the next release of code was built by modifying the previous one. Hence Regression Testing gains significant importance in these situations. The team tests if the newly added functionality works correctly and that the previously released functionality still works as expected. Test Automation also gains importance due to short delivery timelines. Test Automation may prove effective in ensuring that everything that needs to be tested was covered. It is not necessary that costly tools be purchased to automate testing. Test Automation can be achieved in a relatively cost effective way by utilizing the various open source tools or by creating in-house scripts. These scripts can run one or more test cases to exercise a unit of code and verify the results or to test several modules. This would vary with the complexity of the Project and the experience of the Team Typical bugs found when doing agile testing? Although nothing is typical about any Agile Development Project and each project may have its own set of complexities, by the very nature of the paradigm bugs may be introduced in the system when a piece of code is modified/enhanced/changed by one or more Developers. Whenever a piece of code is changed it is possible that bugs have been introduced to it or previously working code is now broken. New bugs/defects can be introduced at every change or old bugs/defects may be reopened. Steps Taken to Effectively Test in Agile Development Methodology: As a wise person once said there is no substitute to hard work. The only way one can effectively test is by ensuring Sufficient Test Coverage and Testing Effort Automated or otherwise. The challenge could be lack of documentation, but the advantage could be close communication between team members thereby resulting in greater clarity of thought and understanding of the system. Each Time Code is changed Regression Testing is done. Test Coverage is ensured by having Automated Scripts and the right mix/combination of People executing the

Test Cases. Exploratory Testing may also be encouraged. Exploratory Tests are not predesigned or pre-defined. The Tests are designed and executed immediately. Similarly ad hoc testing may also be encouraged. Ad-hoc testing is done based on the testers experience and skills. While automated Test Cases will ensure that the Test Cases scripted are executed as defined, the team may not have enough time to design and script all the test cases. Ensuring software test coverage To ensure that delivered product meets the end users requirements it is important that sufficient testing is done and all scenarios are tested. Sufficient Test Coverage in an Agile Development Scenario may be tricky but with close cooperation and the right team dynamics it may not be impossible. The objectives of the Project should be clear to the entire team. Many teams advocate Test Driven Development. At every stage the Software is tested if it meets the Requirements. Every Requirement is translated to a Test Case and Software is validated/ verified. While the Processes and Documentation are not stressed upon sufficient steps are taken to ensure that the software is delivered as per the User Expectations. This implies that each Software delivery should be tested thoroughly before it is released. The timelines being short requires that the person testing the software has sufficient knowledge about the system and its objectives