Orthogonal Array for Testing

color:#3366AA"">Orthogonal Array for Testing

1. INTRODUCTION

The job of a software tester is to try break the
system in every possible way so that all faults are detected, thereby
increasing the likelihood of delivering fault-free software to the customer. In
a fast paced industry, the main challenges to the testing community are: 

• To reduce cycle time of the test phase
• To find maximum defects with minimal test cases
• To offer maximum coverage with minimum test
cases

The testing process/ test strategy should be
efficient and effective to help face these challenges. Orthogonal Array testing
strategy is one statistical method pioneered by Prof Taguchi that helps achieving
a comprehensive and representative suite of test combinations.
2. ORTHOGONAL ARRAY TESTING STRATEGY (OATS)

OATS is a systematic statistical way of testing
pair-wise interactions. It has been found useful in testing combinations of
configurable options (variables).

An orthogonal array has specific properties.
First, an OA is a rectangular array or table of values, presented in rows and
columns. Each column represents a variable or parameter. Here is some
terminology for working with orthogonal arrays:

Runs The number of rows in the array. This
directly translates to the number of test cases that will be generated by the
OATS technique.
Factors The number of columns in an array. This
directly translates to the maximum number of variables that can be handled by
this array.
Levels The maximum number of values that can be
taken on by any single factor. An orthogonal array will contain values from 0
to Levels-1.

Orthogonal arrays are most often named following
the pattern - LRuns (LevelsFactors)
2.1. How Does OATS Help Optimize Testing?

• Creates an optimized test suite with lesser test
cases, and uncovers most of the bugs
• Detects all single mode, double mode, and
multimode faults*.
• Guarantees testing the pair-wise combinations of
all the selected variables.
• Exercises some of the complex combinations of
all the variables.
• Is simpler to generate and is less error prone
than test sets created manually.
• Helps in productivity improvement with cycle
time reduction.
• Is independent of platforms and domains.

* Detect and isolate all single-mode faults: A
single-mode fault is a consistent problem with any level of any single
parameter. For example, if all cases of factor A at Level A1 cause error
conditions, it is a single-mode failure. 
Detect all double-mode faults: If there exists a
consistent problem when specific levels of two parameters occur together, it is
called a double-mode fault. Indeed, a double-mode fault is an indication of
pair-wise incompatibility or harmful interaction between two test parameters. 
Multi-mode faults: Many multimode faults are also
there and can be understood by studying the geometric view of the test cases. 
2.2. How to Use This Technique?

The OATS technique is simple and straightforward
and can be customized based on available time and known problems. The first
step in constructing an orthogonal array to fit a specific case is to calculate
the minimum number of tests that must be performed -- this is also called
degrees of freedom. (The degrees of freedom associated with interaction between
two factors A and B, are given by the product of the degrees of freedom for
each of the two factors, [degrees of freedom for A] x [degrees of freedom for
B])

1. Decide how many independent variables will be
tested for interaction. This will map to the Factors of the array. 
2. Decide the maximum number of values that each
independent variable will take on. This will map to the Levels of the array.
3. Find a suitable orthogonal array with the
smallest number of Runs. (Suitable array is one that has at least as many
Factors as needed and has at least as many levels for each of those factors as
decided in.)
4. Map the Factors and values onto the array.
5. Transcribe the Runs into test cases, adding any
particularly suspicious combinations that are not generated.

For example, assume that we have a system with 5
independent variables (P, Q, R, S, and T). Variables P and Q each have three
possible values (P1-3 and Q1-3). Variables R and S each have four possible
values (R1-4 and S1-4), and variable T has six possible values (T1-6). To test
all of the possible combinations, it would take 864 test cases (3 x 3 x 4 x 4 x
6 = 864). Using an L18(3661) OA can significantly reduce the number of test
cases to just 18 to test all the pair-wise combinations. 
3. CASE STUDIES USING OATS
3.1. Example :

Application that contains a Configure the scheme
section with three main subsections Scheme selection, Start & End date and
Contribution to the fund. The Configure the scheme section also takes an
instance of Investment calculation section with subsections Investment Amount,
Number of units and Term of retirement. 

To exhaustively test all combinations of the above
sections in Administrator Application, it would take 27 test cases (for
validating the three sections with three subsections). In real time project
scenario, the assigned representative has developed approximately 26 test cases
to test this functionality. Average time taken for each test case generation is
20 minutes. Also, this interaction is probably a very small portion of the
entire system being tested. 
Applying OATS: 
1. Three independent variables are selected for
interaction. These are the Factors of the array. 
2. A maximum of three values exist that each
independent variable can take on.
3. An L9 (34) orthogonal array is decided for this
testing. 

An L9 (34) Orthogonal array – 
(Where three levels of each factor is represented
as 0, 1, 2)
Factor 1 Factor 2 Factor 3 Factor 4
Run 1 0 0 0 0
Run 2 0 1 1 2
Run 3 0 2 2 1
Run 4 1 0 1 1
Run 5 1 1 2 0
Run 6 1 2 0 2
Run 7 2 0 2 2
Run 8 2 1 0 1
Run 9 2 2 1 0

4. When mapping the factors and values onto the
array,

An L9 (34) orthogonal array – 
(Where three levels of each factor is represented
as 0, 1, 2)
Configure the scheme (A) Issuing the policy – (B)
Investment calculation – (C) Factor 4
TC-1 A0 B0 C0
TC- 2 A0 B1 C1
TC- 3 A0 B2 C2
TC- 4 A1 B0 C1
TC- 5 A1 B1 C2
TC- 6 A1 B2 C0
TC- 7 A2 B0 C2
TC- 8 A2 B1 C0
TC- 9 A2 B2 C1

5. There are no "left over" Levels;
however there was an extra Factor in the original array. This factor can simply
be ignored; it does not change the properties of the test set generated from
the array. 
6. Taking the test case values from each run,
there should be nine test cases. Any particularly suspicious combinations that
are not generated can be included additionally.

Results: OATS technique significantly reduces the
number of test cases to 9 in this project, thereby considerably reducing
generation, execution, and overall cycle time by around 65%. Also, in
traditional method the coverage is mapped on the basis of Bidirectional
Traceability Matrix (BDTM), and this might involve the chance of missing some
of the combinations. But orthogonal array based testing employs the method of
pair wise combinations and maximizes the coverage. 

4. CONCLUSION

Orthogonal array based testing considerably
reduces the overall test cycle time and product cycle time. This method is
found effective in arriving at an efficient and effective test strategy,
producing an optimum suit of representative but exhaustive combination of test
cases, and optimizing the productivity.
5. REFERENCES

• Phadke, M.S. "Planning Efficient Software
Tests" CrossTalk Journal of Defense Software Engineering
• Taguchi, Genichi. "System of Experimental
Design" Edited by Don Clausing. New
York: UNIPUB/Krass International Publications, Volume
1 & 2, 1987
• Jeremy M. Harrell, “Orthogonal Array Testing
Strategy (OATS) Technique” Seilevel, Inc
• Sloane, Neil J. A. A Library of Orthogonal
Arrays. Information
Sciences Research
Center
•
http://softwaretesting-guide.blogspot.com/2008/03/245-orthogonal-array-testing.html