A survey on code coverage as a stopping criterion for unit testing (original) (raw)
Related papers
A Study on Test Coverage in Software Testing
ipcsit.net
Abstract: Test Coverage is an important indicator of software quality and an essential part of software maintenance. It helps in evaluating the effectiveness of testing by providing data on different coverage items. Although much research effort has been put on how to get coverage ...
Test case quality: an empirical study on belief and evidence
arXiv (Cornell University), 2023
Software testing is a mandatory activity in any serious software development process, as bugs are a reality in software development. This raises the question of quality: good tests are effective in finding bugs, but until a test case actually finds a bug, its effectiveness remains unknown. Therefore, determining what constitutes a good or bad test is necessary. This is not a simple task, and there are a number of studies that identify different characteristics of a good test case. A previous study evaluated 29 hypotheses regarding what constitutes a good test case, but the findings are based on developers' beliefs, which are subjective and biased. In this paper we investigate eight of these hypotheses, through an extensive empirical study based on open software repositories. Despite our best efforts, we were unable to find evidence that supports these beliefs. This indicates that, although these hypotheses represent good software engineering advice, they do not necessarily mean that they are enough to provide the desired outcome of good testing code.
Code Coverage for Suite Evaluation by Developers
2014
One of the key concerns of developers testing code is howto determine a test suite’s quality – its ability to find faults.The most common approach in industry is to use code coverage as a measure for test suite quality, and diminishing returns in coverage or high absolute coverage as a stopping rule. In testing research, suite quality is often evaluated by measuring a suite’s ability to kill mutants, which are artificially seeded potential faults. Mutation testing is effective but expensive, thus seldom used by practitioners. Deter-mining which criteria best predict mutation kills is therefore critical to practical estimation of test suite quality. Previous work has only used small sets of programs, and usually compares multiple suites for a single program. Practitioners, however, seldom compare suites — they evaluate one suite. Using suites (both manual and automatically generated) from a large set of real-world open-source projects shows that results for evaluation differ from those for suite-comparison: statement coverage (not block, branch, or path)predicts mutation kills best.
Lessons learned from conducting industry surveys in software testing
2013 1st International Workshop on Conducting Empirical Studies in Industry (CESI), 2013
We have conducted a number of industry studies looking into aspects of software testing. These include a survey of practitioner view on software tester performance indicators and team formation issues; detailed worklog analysis of representative tasks conducted by software testers; feedback on a proposed approach to software tester performance evaluation; and collecting indicative personality factors of software testers vs other software developers. These have proved to be challenging exercises. Key issues to overcome include reaching appropriate testers in industry, gaining management and company consent, structuring surveys and questionnaires to balance time vs detail, and gaining sufficient response rate and response quality to be useful.
Software Testing: The State of the Practice
IEEE Software, 2017
A web-based survey examined how software professionals used testing. The results offer opportunities for further interpretation and comparison to software testers, project managers, and researchers. The data includes characteristics of practitioners, organizations, projects, and practices.
Open Problems in Software Test Coverage
Lecture Notes on Software Engineering, 2014
Software testing is concerned with the production and maintenance of high quality systems. Test coverage is one major component in software testing. It gives a meaningful description of how much testing is required. The level of test coverage is an indicator of testing thoroughness, which in turn helps testers decide when to stop testing. Several studies addressed coverage criteria effectiveness and adequacy. However, there are several interesting open problems. We identify nine open problems in test coverage and discuss what issues need to be addressed. The objective of this paper is to direct the attention of researchers and testers toward the areas that need to be investigated.
IEICE Transactions on Information and Systems, 2011
Kazunori SAKAMOTO †a) , Fuyuki ISHIKAWA † †b) , Hironori WASHIZAKI †c) , and Yoshiaki FUKAZAWA †d) , Members SUMMARY Test coverage is an important indicator of whether software has been sufficiently tested. However, there are several problems with the existing measurement tools for test coverage, such as their cost of development and maintenance, inconsistency, and inflexibility in measurement. We propose a consistent and flexible measurement framework for test coverage that we call the Open Code Coverage Framework (OCCF). It supports multiple programming languages by extracting the commonalities from multiple programming languages using an abstract syntax tree to help in the development of the measurement tools for the test coverage of new programming languages. OCCF allows users to add programming language support independently of the test-coverage-criteria and also to add test-coverage-criteria support independently of programming languages in order to take consistent measurements in each programming language. Moreover, OCCF provides two methods for changin the measurement range and elements using XPath and adding user code in order to make more flexible measurements. We implemented a sample tool for C, Java, and Python using OCCF. OCCF can measure four test-coveragecriteria. We also confirmed that OCCF can support C#, Ruby, JavaScript, and Lua. Moreover, we reduced the lines of code (LOCs) required to implement measurement tools for test coverage by approximately 90% and the time to implement a new test-coverage-criterion by over 80% in an experiment that compared OCCF with the conventional non-framework-based tools.
A Survey on Unit Testing Practices and Problems
2014 IEEE 25th International Symposium on Software Reliability Engineering, 2014
Unit testing is a common practice where developers write test cases together with regular code. Automation frameworks such as JUnit for Java have popularised this approach, allowing frequent and automatic execution of unit test suites. Despite the appraisals of unit testing in practice, software engineering researchers see potential for improvement and investigate advanced techniques such as automated unit test generation. To align such research with the needs of practitioners, we conducted a survey amongst 225 software developers, covering different programming languages and 29 countries, using a global online marketing research platform. The survey responses confirm that unit testing is an important factor in software development, and suggest that there is indeed potential and need for research on automation of unit testing. The results help us to identify areas of importance on which further research will be necessary (e.g., maintenance of unit tests), and also provide insights into the suitability of online marketing research platforms for software engineering surveys.
Effect of code coverage on software reliability measurement
2001
& Conclusions-Existing software reliability-growth models often overestimate the reliability of a given program. Empirical studies suggest that the over-estimations exist because the models do not account for the nature of the testing. Every testing technique has a limit to its ability to reveal faults in a given system. Thus, as testing continues in its region of saturation, no more faults are discovered and inaccurate reliability-growth phenomena are predicted from the models. This paper presents a technique intended to solve this Problem, using both time & code coverage measures for the prediction of software failures in operation. Coverage information collected during testing is used only to consider the effective portion of the test data. Execution time between test cases, which neither increases code coverage nor causes a failure, is reduced by a parameterized factor. Experiments were conducted to evaluate this technique, on a program created in a simulated environment with simulated faults, and on two industrial systems that contained tenths of ordinary faults. Two well-known reliability models, Goel-Okumoto and Musa-Okumoto, were applied to both the raw data and to the data adjusted using this technique. Results show that overestimation of reliability is properly corrected in the cases studied. This new approach has potential, not only to achieve more accurate applications of software reliability models, but to reveal effective ways of conducting software testing.