Cross-Entropy Based Testing (original) (raw)
Related papers
Efficient Software Verification: Statistical Testing Using Automated Search
IEEE Transactions on Software Engineering, 2010
Statistical testing has been shown to be more efficient at detecting faults in software than other methods of dynamic testing such as random and structural testing. Test data are generated by sampling from a probability distribution chosen so that each element of the software's structure is exercised with a high probability. However, deriving a suitable distribution is difficult for all but the simplest of programs. This paper demonstrates that automated search is a practical method of finding near-optimal probability distributions for real-world programs, and that test sets generated from these distributions continue to show superior efficiency in detecting faults in the software.
A Divergence-Oriented Approach to Adaptive Random Testing of Java Programs
2009
Adaptive Random Testing (ART) is a testing technique which is based on an observation that a test input usually has the same potential as its neighbors in detection of a specific program defect. ART helps to improve the efficiency of random testing in that test inputs are selected evenly across the input spaces. However, the application of ART to objectoriented programs (e.g., C++ and Java) still faces a strong challenge in that the input spaces of object-oriented programs are usually high dimensional, and therefore an even distribution of test inputs in a space as such is difficult to achieve. In this paper, we propose a divergence-oriented approach to adaptive random testing of Java programs to address this challenge. The essential idea of this approach is to prepare for the tested program a pool of test inputs each of which is of significant difference from the others, and then to use the ART technique to select test inputs from the pool for the tested program. We also develop a tool called ARTGen to support this testing approach, and conduct experiment to test several popular opensource Java packages to assess the effectiveness of the approach. The experimental result shows that our approach can generate test cases with high quality.
ARTOO: adaptive random testing for object-oriented software
2008
Intuition is often not a good guide to know which testing strategies will work best. There is no substitute for experimental analysis based on objective criteria: how many faults a strategy finds, and how fast. "Random" testing is an example of an idea that intuitively seems simplistic or even dumb, but when assessed through such criteria can yield better results than seemingly smarter strategies. The efficiency of random testing is improved if the generated inputs are evenly spread across the input domain. This is the idea of Adaptive Random Testing (ART).
An evaluation of random testing
Software Engineering, IEEE …, 1984
Random testing of programs has usually (but not always) evaluate how well a set of programs is tested by random testbeen viewed as a worst case of program testing. Testing strategies that ing by using a variety of coverage measures such as the proportake into account the program structure are generally preferred. Path tion of segments and branches executed by randomly generated testing is an often proposed ideal for structural testing. Path testing is treated here as an instance of partition testing, where by partition test-sets of test cases. ing is meant any testing scheme which forces execution of at least one test case from each subset of a partition of the input domain. Simulation results are presented which suggest that random testing may often Let 0 be the probability that a program will fail to execute be more cost effective than partition testing schemes. Also, results of correctly on an input case chosen from a given input distribuactual random testing experiments are presented which confirm the tion. If the program is used for a long period of time with viability of random testing as a useful validation tool.
Enhanced Random Testing for Programs with High Dimensional Input Domains
2007
Random Testing (RT) is a fundamental technique of software testing. Adaptive Random Testing (ART) has recently been developed as an enhancement of RT that has better fault detection effectiveness. Several methods (algorithms) have been developed to implement ART. In most ART algorithms, however, the above enhancement diminishes when the dimensionality of the input domain increases. In this paper, we investigate the nature of failure regions in high dimensional input domains and propose enhanced random testing algorithms that improve the fault detection effectiveness of RT in high dimensional input domains.
Software Quality Journal, 2008
Adaptive random testing (ART), an enhancement of random testing (RT), aims to both randomly select and evenly spread test cases. Recently, it has been observed that the effectiveness of some ART algorithms may deteriorate as the number of program input parameters (dimensionality) increases. In this article, we analyse various problems of one ART algorithm, namely fixed-sized-candidate-set ART (FSCS-ART), in the high dimensional input domain setting, and study how FSCS-ART can be further enhanced to address these problems. We propose to add a filtering process of inputs into FSCS-ART to achieve a more even-spread of test cases and better failure detection effectiveness in high dimensional space. Our study shows that this solution, termed as FSCS-ART-FE, can improve FSCS-ART not only in the case of high dimensional space, but also in the case of having failure-unrelated parameters. Both cases are common in real life programs. Therefore, we recommend using FSCS-ART-FE instead of FSCS-ART whenever possible. Other ART algorithms may face similar problems as FSCS-ART; hence our study also brings insight into the improvement of other ART algorithms in high dimensional space.
Automated testing of stochastic systems
Proceedings of the 2006 international symposium on Software testing and analysis, 2006
Automated tests can play a key role in ensuring system quality in software development. However, significant problems arise in automating tests of stochastic algorithms. Normally, developers write tests that simply check whether the actual result is equal to the expected result (perhaps within some tolerance). But for stochastic algorithms, restricting ourselves in this way severely limits the kinds of tests we can write: either to trivial tests, or to fragile and hard-tounderstand tests that rely on a particular seed for a random number generator. A richer and more powerful set of tests is possible if we accommodate tests of statistical properties of the results of running an algorithm many times. The work described in this paper has been done in the context of a real-world application, a large-scale simulation of urban development designed to inform major decisions about land use and transportation. We describe our earlier experience with using automated testing for this system, in which we took a conventional approach, and the resulting difficulties. We then present a statistically based approach for testing stochastic algorithms based on hypothesis testing. Three different ways of constructing such tests are given, which cover the most commonly used distributions. We evaluate these tests in terms of frequency of failing when they should and when they should not, and conclude with guidelines and practical suggestions for implementing such unit tests for other stochastic applications.
Path-oriented random testing through iterative partitioning (IP-PRT)
TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES
Path-oriented random testing aims at generating a uniformly distributed sequence of test data from a program input domain space to traverse a desired execution path of the program. To this aim, this article proposes a new algorithm to refine a program inputs domain space from invalid subdomains not covering the path. The validity of the subdomains is checked by a constraint propagation method against the path constraints (PCs). The proposed algorithm uses a divideand-conquer technique to iteratively split the inputs domain into subdomains and each time refutes those subdomains that are inconsistent with the PCs. The remaining shrunken subdomains provide all possible test data covering the desired path. Obviously, the more accurate the input domain is, the more effective test data will result. Experiments show the proposed method outperformed other related methods on a set of classical benchmark programs.
Feedback-Directed Random Test Generation
International Conference on Software Engineering, 2007
We present a technique that improves random test generation by incorporating feedback obtained from executing test inputs as they are created. Our technique builds inputs incrementally by randomly selecting a method call to apply and finding arguments from among previously-constructed inputs. As soon as an input is built, it is executed and checked against a set of contracts and filters. The result of the execution determines whether the input is redundant, illegal, contract-violating, or useful for generating more inputs. The technique outputs a test suite consisting of unit tests for the classes under test. Passing tests can be used to ensure that code contracts are preserved across program changes; failing tests (that violate one or more contract) point to potential errors that should be corrected. Our experimental results indicate that feedback-directed random test generation can outperform systematic and undirected random test generation, in terms of coverage and error detectio...
Enhancing Performance of Random Testing through Markov Chain Monte Carlo Methods
IEEE Transactions on Computers, 2000
In this paper, we propose a probabilistic approach to finding failurecausing inputs based on Bayesian estimation. According to our probabilistic insights of software testing, the test case generation algorithms are developed by Markov chain Monte Carlo (MCMC) methods. Dissimilar to existing random testing schemes such as adaptive random testing, our approach can also utilize the prior knowledge on software testing. In experiments, we compare effectiveness of our MCMC-based random testing with both ordinary random testing and adaptive random testing in real program sources. These results indicate the possibility that MCMC-based random testing can drastically improve the effectiveness of software testing.