Automatic calculation of coverage profiles for coverage-based testings (original) (raw)
Related papers
Extending Coverage Criteria by Evaluating Their Robustness to Code Structure Changes
2012
Code coverage is usually used as a measurement of testing quality and as adequacy criterion. Unfortunately, code coverage is very sensitive to modifications of the code structure, and, therefore, the same test suite can achieve different degrees of coverage on the same program written in two syntactically different ways. For this reason, code coverage can provide the tester with misleading information.
2007
Embedded systems are often used in safety-critical environments. Thus, thorough testing of them is mandatory. A quite active research area is the automatic test-case generation for testing embedded systems. To achieve high retargetability of the testing framework, the test-case generation has to be done at source-code level. However, it is challenging to guarantee that the test-cases obtained from the source code are also valid at the object-code level, since even in safety-critical domains programs are optimized during compilation, i.e., the compiler may also introduce new control-flow decisions to the program.
Guided test generation for coverage criteria
2010 IEEE International Conference on Software Maintenance, 2010
Test coverage criteria including boundary-value and logical coverage such as Modified Condition/Decision Coverage (MC/DC) have been increasingly used in safety-critical or mission-critical domains, complementing those more popularly used structural coverage criteria such as block or branch coverage. However, existing automated test-generation approaches often target at block or branch coverage for test generation and selection, and therefore do not support testing against boundary-value coverage or logical coverage. To address this issue, we propose a general approach that uses instrumentation to guide existing test-generation approaches to generate test inputs that achieve boundary-value and logical coverage for the program under test. Our preliminary evaluation shows that our approach effectively helps an approach based on Dynamic Symbolic Execution (DSE) to improve boundary-value and logical coverage of generated test inputs. The evaluation results show 30.5% maximum (23% average) increase in boundary-value coverage and 26% maximum (21.5% average) increase in logical coverage of the subject programs under test using our approach over without using our approach. In addition, our approach improves the fault-detection capability of generated test inputs by 12.5% maximum (11% average) compared to the test inputs generated without using our approach.
A Comparative Evaluation of Test Coverage Techniques Effectiveness
Journal of Software Engineering and Applications
Software systems have become complex and challenging to develop and maintain because of the large size of test cases with increased scalability issues. Test case prioritization methods have been successfully utilized in test case management. However, the prohibitively exorbitant cost of large test cases is now the mainstream in the software industry. The growth of agile test-driven development has increased the expectations for software quality. Yet, our knowledge of when to use various path testing criteria for costeffectiveness is inadequate due to the inherent complexity in software testing. Existing researches attempted to address the issue without effectively tackling the scalability of large test suites to reduce time in regression testing. In order to provide a more accurate way of fault detection in software projects, we introduced novel coverage criteria, called Incremental Cluster-based test case Prioritization (ICP), and investigated its potentials by making a comparative evaluation with three un-clustered traditional coverage-based criteria: Prime-Path Coverage (PPC), Edge-Pair Coverage (EPC) and Edge Coverage (EC) based on mutation analysis. By clustering test suites, based on their dynamic run-time behavior, the number of pair-wise comparisons is reduced significantly. To compare, we analyzed 20 functions from 25 C programs, instrumented faults into the programs, and used the Mull mutation tool to generate mutants and perform a statistical analysis of the results. The experimental results show that ICP can lead to cost-effective improvements in fault detection.
The Study of Various Code Coverage Tools
International Journal of Computer Trends and Technology, 2014
Code coverage is used to describe the degree to which the source code of a program is tested. There are many code coverage testing tools available, working on different criteria providing different features. Here we have studied five code coverage tools and out of which one tool was actually evaluated for their proposed features. A comparative study is presented on the basis of the set criteria. Keywords— software testing, code coverage, code coverage tools, code coverage criteria.
Using mutation analysis for assessing and comparing testing coverage criteria
IEEE Transactions on Software Engineering, 2006
The empirical assessment of test techniques plays an important role in software testing research. One common practice is to instrument faults in subject software, either manually or by using a program that generates all possible mutants based on a set of mutation operators. The latter allows the systematic, repeatable seeding of large numbers of faults, thus facilitating the statistical analysis of fault detection effectiveness of test suites; however, we do not know whether empirical results obtained this way lead to valid, representative conclusions. Focusing on four common control and data flow criteria (Block, Decision, C-use, P-use), this paper investigates this important issue based on a middle size industrial program with a comprehensive pool of test cases and known faults. Based on the data available thus far, the results are very consistent across the investigated criteria, as they show that the use of mutation operators is yielding trustworthy results: generated mutants can be used to predict the detection effectiveness of real faults. Applying such mutation analysis, we then investigate the relative cost and effectiveness of the above mentioned criteria by revisiting fundamental questions regarding the relationships between fault detection, test suite size, and control/data flow coverage. Although such questions have been partially investigated in previous studies, we can use a large number of mutants which helps decrease the impact of random variation in our analysis and allows us to use a different analysis approach. Our results are then compared with published studies, plausible reasons for the differences are provided, and this leads us to suggest a way to tune the mutation analysis process to possible differences in fault detection probabilities in a specific environment.
Automated Test Case Generation Based on Coverage Analysis
2009 Third IEEE International Symposium on Theoretical Aspects of Software Engineering, 2009
We present a tool for the automated generation of unit tests. It symbolically executes Java bytecode in order to find execution paths through a program. To efficiently accomplish this task, it uses constraint solving, choice-point generation and backtracking. As the number of test cases found might be very high and most of them are redundant, we propose a novel way to eliminate test cases based on their contribution to the global coverage of the controlflow and data-flow. Besides discussing the techniques used to achieve this, we present experimental results to prove the feasibility of our approach.
Reinforced Condition/Decision Coverage (RC/DC): A New Criterion for Software Testing
ZB2002: Formal Specification and Development in Z and B, 2nd International Conference of B and Z Users, 2002
A new Reinforced Condition/Decision Coverage (RC/DC) criterion for software testing is proposed. This criterion provides further development of the well-known Modified Condition/Decision Coverage (MC/DC) criterion and is more suitable for testing of safety-critical software. Formal definitions in the Z notation for RC/DC, as well as MC/DC, are presented. Specific examples of using of these criteria are considered and some features are formally proved.