Test Coverage and Traceability

Traceability:
Traceability is a term that refers to the ability to review all the test cases you have defined for each requirement specification. Traceability also includes the ability to review any defects that have been associated with failed test cases. By reviewing traceability, you can ensure that each requirement has an adequate number (and level) of testing, and you can determine which requirements spawned the most defects during the testing process.

Requirements Review:
Quality starts with requirements definition. Upon defining a requirements specification, it is important to have a team review of the requirement specification. The owner of the requirement specification should distribute the requirement specification to all team members for review (this includes the project manager, analyst that created the specification, the programmer(s) that will perform the coding, and the tester(s) that is responsible for testing the new requirement). Prior to the meeting, each person should have already reviewed the requirement in detail and should come prepared with comments for the review. Below are things that should be reviewed:

1. Completeness - Is the requirement defined in enough detail to code and test against?
2. Accuracy - Is the requirement accurately defined and logical? Are there missing elements?
3. Testable - Is the requirement testable (can a complete set of test cases be written for it)?

Test Case Review:
Once the requirement specification has been reviewed and approved by the reviewers, the test team should begin developing test cases. As each test case is created, it should be identified as belonging to the requirement specification. This ensures that you can quickly determine exactly what test cases belong to each specification, which provides traceability. Once all test cases have been defined for the requirement specification, the tester should distribute the test cases to all team members for review (the same people that attended the requirements review). Like the requirements review, the reviewers should review the test cases in detail and should come prepared with comments for the review. Below are the things that should be reviewed:

1. Positive Test Cases - Positive test cases are designed to test the feature per the design. It does not try to break the feature, it is simply test cases that demonstrate how a user might use the software in a normal environment. The review should ensure that there are an adequate number of test cases for this.

2. Negative Test Cases - Negative test cases are designed to test the feature in ways that the developer may not have envisioned. This includes bounds and logical tests. For example, try testing invalid dates, entering alphabetic information in numeric fields, entering numeric information outside of the bounds that it was designed for (e.g. try entering 101% in a percentage field that should only allow up to 100%), and entering field values that are larger than the field size (e.g. try entering 101 characters in a field that is designed for 100 characters). The review should ensure that there are an adequate number of test cases for this.

3. Performance - Performance test cases ensure that the code will not become unusable with large amounts of data. For example, import 50,000 items and record the timings. Compare those timings to when you only have 50 items. For most applications, acceptable response time is anything under 5 seconds, while good response time is anything under 2 seconds. This may vary depending on the application and your own performance guidelines. The review should ensure that there are an adequate number of test cases for this.

4. Security Testing - If the feature is a secured feature, there should be security test cases to ensure that the correct rights are granted before specific actions can occur. The review should ensure that there are an adequate number of test cases for this.

5. Regression Testing - If the feature will become a base part of your product, identify specific test cases from this test suite that would be a good candidate for Regression Testing in future releases. The review should ensure that there are an adequate number of test cases for this.

Defect Linkage:
Once testing commences, it is important to identify the defects that were created based on the specific test case that failed. Good defect management solutions will allow you to automatically generate a defect when a test case fails and provide a linkage between the failed test case and the automatically generated defect. Assuming your test cases are linked back to the original requirement specification, you can then link the defects by requirement. This allows you to determine which requirements spawn the most defects.