iTree: Efficiently Discovering High-Coverage Configurations Using Interaction Trees (original) (raw)
Related papers
Test Case-Aware Combinatorial Interaction Testing
IEEE Transactions on Software Engineering, 2013
The configuration spaces of modern software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations by using a battery of test cases. Traditional covering arrays, while taking system-wide inter-option constraints into account, do not provide a systematic way of handling test case-specific inter-option constraints. The basic justification for t-way covering arrays is that they can cost-effectively exercise all system behaviors caused by the settings of t or fewer options. In this work we hypothesize, however, that in the presence of test case-specific inter-option constraints, many such behaviors may not be tested due to masking effects caused by the overlooked test case-specific constraints. For example, if a test case refuses to run in a configuration due to an unsatisfied test case-specific constraint, none of the valid option setting combinations appearing in the configuration will be tested by that test case. To account for test case-specific constraints, we introduce a new combinatorial object, called a test case-aware covering array. A t-way test case-aware covering array is not just a set of configurations as is the case in traditional covering arrays, but a set of configurations, each of which is associated with a set of test cases, such that all test case-specific constraints are satisfied and that, for each test case, each valid combination of option settings for every combination of t options appears at least once in the set of configurations that the test case is associated with. We furthermore present three algorithms to compute test case-aware covering arrays. Two of the algorithms aim to minimize the number of configurations required (one is fast, but produces larger arrays, the other is slower, but produces smaller arrays), whereas the remaining algorithm aims to minimize the number of test runs required. The results of our empirical studies conducted on two widely-used highly-configurable software systems suggest that test case-specific constraints do exist in practice, that traditional covering arrays suffer from masking effects caused by the ignorance such constraints, and that test case-aware covering arrays are better than other approaches in handling test case-specific constraints, thus avoiding masking effects.
Measuring and Specifying Combinatorial Coverage of Test Input Configurations
Innovations in systems and software engineering, 2016
A key issue in testing is how many tests are needed for a required level of coverage or fault detection. Estimates are often based on error rates in initial testing, or on code coverage. For example, tests may be run until a desired level of statement or branch coverage is achieved. Combinatorial methods present an opportunity for a different approach to estimating required test set size, using characteristics of the test set. This paper describes methods for estimating the coverage of, and ability to detect, t-way interaction faults of a test set based on a covering array. We also develop a connection between (static) combinatorial coverage and (dynamic) code coverage, such that if a specific condition is satisfied, 100% branch coverage is assured. Using these results, we propose practical recommendations for using combinatorial coverage in specifying test requirements.
Code-Aware Combinatorial Interaction Testing
IET Software, 2019
Combinatorial interaction testing (CIT) is a useful testing technique to address the interaction of input parameters in software systems. In many applications, the technique has been used as a systematic sampling technique to sample the enormous possibilities of test cases. In the last decade, most of the research activities focused on the generation of CIT test suites as it is a computationally complex problem. Although promising, less effort has been paid for the application of CIT. In general, to apply the CIT, practitioners must identify the input parameters for the Software-under-test (SUT), feed these parameters to the CIT tool to generate the test suite, and then run
iGen: dynamic interaction inference for configurable software
Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering - FSE 2016, 2016
To develop, analyze, and evolve today's highly configurable software systems, developers need deep knowledge of a system's configuration options, e.g., how options need to be set to reach certain locations, what configurations to use for testing, etc. Today, acquiring this detailed information requires manual effort that is difficult, expensive, and error prone. In this paper, we propose iGen, a novel, lightweight dynamic analysis technique that automatically discovers a program's interactions-expressive logical formulae that give developers rich and detailed information about how a system's configuration option settings map to particular code coverage. iGen employs an iterative algorithm that runs a system under a small set of configurations, capturing coverage data; processes the coverage data to infer potential interactions; and then generates new configurations to further refine interactions in the next iteration. We evaluated iGen on 29 programs spanning five languages; the breadth of this study would be unachievable using prior interaction inference tools. Our results show that iGen finds precise interactions based on a very small fraction of the number of possible configurations. Moreover, iGen's results confirm several earlier hypotheses about typical interaction distributions and structures. CCS Concepts •Software and its engineering → Software testing and debugging; Software configuration management and version control systems; Dynamic analysis;
SPLat: lightweight dynamic analysis for reducing combinatorics in testing configurable systems
Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, 2013
Many programs can be configured through dynamic and/or static selection of configuration variables. A software product line (SPL), for example, specifies a family of programs where each program is defined by a unique combination of features. Systematically testing SPL programs is expensive as it can require running each test against a combinatorial number of configurations. Fortunately, a test is often independent of many configuration variables and need not be run against every combination. Configurations that are not required for a test can be pruned from execution. This paper presents SPLat, a new way to dynamically prune irrelevant configurations: the configurations to run for a test can be determined during test execution by monitoring accesses to configuration variables. SPLat achieves an optimal reduction in the number of configurations and is lightweight compared to prior work that used static analysis and heavyweight dynamic execution. Experimental results on 10 SPLs written in Java show that SPLat substantially reduces the total test execution time in many cases. Moreover, we demonstrate the scalability of SPLat by applying it to a large industrial code base written in Ruby on Rails.
Combinatorial Interaction Testing Using Test Case Based Constraints
International Journal of Engineering Development and Research (IJEDR), ISSN:2321-9939, 2014
The bigger applications like web servers e.g. Apache, databases e.g. mysql, and application servers e.g. Tomcat are required to be customizable to adapt to particular runtime contexts and application scenarios. One way to support software customization is to provide configuration options through which the behavior of the system can be controlled. But the configuration spaces of modern software systems are too large to test exhaustively. The proposed method called traditional combinatorial interaction testing which samples the covering arrays and the test cases to make the highly configured application or system. In the combinatorial interaction testing generally we generate configuration options and then we apply test cases for each configuration options. That causes masking effect or skipping of the some reliable configuration options. The obtained system is towards to highly configurable system which uses traditional combinatorial Interaction Testing. Traditional Combinatorial Interaction Testing generates test cases with configuration options and uses test case specific constraints and seeding which avoids the masking effect.
Feedback driven adaptive combinatorial testing
Proceedings of the 2011 International Symposium on Software Testing and Analysis - ISSTA '11, 2011
The configuration spaces of modern software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations. The basic justification for CIT approaches is that they can cost-effectively exercise all system behaviors caused by the settings of t or fewer options. We conjecture, however, that in practice many such behaviors are not actually tested because of masking effects-failures that perturb execution so as to prevent some behaviors from being exercised. In this work we present a feedback-driven, adaptive, combinatorial testing approach aimed at detecting and working around masking effects. At each iteration we detect potential masking effects, heuristically isolate their likely causes, and then generate new covering arrays that allow previously masked combinations to be tested in the subsequent iteration. We empirically assess the effectiveness of the proposed approach on two large widely used open source software systems. Our results suggest that masking effects do exist and that our approach provides a promising and efficient way to work around them.
T‐wise combinatorial interaction test suites construction based on coverage inheritance
2011
SUMMARY Combinatorial interaction testing (CIT) is a testing technique that requires covering all t-sized tuples of values out of n parameter attributes or properties modelled after the input parameters or the configuration domain of a system under test. CIT test suites have shown to be very effective in software testing already at pairwise (t= 2) level, and the effectiveness of CIT grows with the tuple width t. Unfortunately, the number of tuples to be tested also does grow.
Towards an Automated Unified Framework to Run Applications for Combinatorial Interaction Testing
Proceedings of the Evaluation and Assessment on Software Engineering, 2019
Combinatorial interaction testing (CIT) is a well-known technique, but industrial experience is needed to determine its efectiveness in diferent application domains. We present a case study introducing a unifed framework for generating, executing and verifying CIT test suites, based on the open-source Avocado test framework. In addition, we present a new industrial case study to demonstrate the efectiveness of the framework. This evaluation showed that the new framework can generate, execute, and verify efective combinatorial interaction test suites for detecting confguration failures (invalid confgurations) in a virtualization system.
Search Based Software Engineering, 2016
The appeal of highly-configurable software systems lies in their adaptability to users' needs. Search-based Combinatorial Interaction Testing (CIT) techniques have been specifically developed to drive the systematic testing of such highly-configurable systems. In order to apply these, it is paramount to devise a model of parameter configurations which conforms to the software implementation. This is a non-trivial task. Therefore, we extend traditional search-based CIT by devising 4 new testing policies able to check if the model correctly identifies constraints among the various software parameters. Our experiments show that one of our new policies is able to detect faults both in the model and the software implementation that are missed by the standard approaches.