Comparing static bug finders and statistical prediction (original) (raw)
Related papers
Software defect prediction using static code metrics underestimates defect-proneness
International Symposium on Neural Networks, 2010
Many studies have been carried out to predict the presence of software code defects using static code metrics. Such studies typically report how a classifier performs with real world data, but usually no analysis of the predictions is carried out. An analysis of this kind may be worthwhile as it can illuminate the motivation behind the predictions and the severity
Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering - ASE 2012, 2012
Software defects can cause much loss. Static bug-finding tools are believed to help detect and remove defects. These tools are designed to find programming errors; but, do they in fact help prevent actual defects that occur in the field and reported by users? If these tools had been used, would they have detected these field defects, and generated warnings that would direct programmers to fix them? To answer these questions, we perform an empirical study that investigates the effectiveness of state-of-the-art static bug finding tools on hundreds of reported and fixed defects extracted from three open source programs: Lucene, Rhino, and As-pectJ. Our study addresses the question: To what extent could field defects be found and detected by state-of-the-art static bug-finding tools? Different from past studies that are concerned with the numbers of false positives produced by such tools, we address an orthogonal issue on the numbers of false negatives. We find that although many field defects could be detected by static bug finding tools, a substantial proportion of defects could not be flagged. We also analyze the types of tool warnings that are more effective in finding field defects and characterize the types of missed defects.
Automated Software Engineering, 2014
Software defects can cause much loss. Static bug-finding tools are designed to detect and remove software defects and believed to be effective. However, do such tools in fact help prevent actual defects that occur in the field and reported by users? If these tools had been used, would they have detected these field defects, and generated warnings that would direct programmers to fix them? To answer these questions, we perform an empirical study that investigates the effectiveness of five state-of-the-art static bug-finding tools (FindBugs, JLint, PMD, CheckStyle, and JCSC) on hundreds of reported and fixed defects extracted from three open source programs (Lucene, Rhino, and AspectJ). Our study addresses the question: To what extent could field defects be detected by state-of-the-art static bug-finding tools? Different from past studies that are concerned with the numbers of false positives produced by such tools, we address an orthogonal issue on the numbers of false negatives. We
Predicting Software Flaws with Low Complexity Models based on Static Analysis Data
Journal of Information Systems Engineering & Management
Due to the constant evolution of technology, each day brings new programming languages, development paradigms, and ways of evaluating processes. This is no different with source code metrics, where there is always new metric classes. To use a software metric to support decisions, it is necessary to understand how to perform the metric collection, calculation, interpretation, and analysis. The tasks of collecting and calculating source code metrics are most often automated, but how should we monitor them during the software development cycle? Our research aims to assist the software engineer to monitor metrics of vulnerability threats present in the source code through a reference prediction model, considering that real world software have non-functional security requirements, which implies the need to know how to monitor these requirements during the software development cycle. As a first result, this paper presents an empirical study on the evolution of the Linux project. Based on static analysis data, we propose low complexity models to study flaws in the Linux source code. About 391 versions of the project were analyzed by mining the official Linux repository using an approach that can be reproduced to perform similar studies. Our results show that it is possible to predict the number of warnings triggered by a static analyzer for a given software project revision as long as the software is continuously monitored.
On the Value of Static Analysis for Fault Detection in Software
IEEE Transactions on Software Engineering, Vol. 32, No. 4, pp. 240-253, 2006
No single software fault-detection technique is capable of addressing all fault-detection concerns. Similarly to software reviews and testing, static analysis tools (or automated static analysis) can be used to remove defects prior to release of a software product. To determine to what extent automated static analysis can help in the economic production of a high-quality product, we have analyzed static analysis faults and test and customer-reported failures for three large-scale industrial software systems developed at Nortel Networks. The data indicate that automated static analysis is an affordable means of software fault detection. Using the Orthogonal Defect Classification scheme, we found that automated static analysis is effective at identifying Assignment and Checking faults, allowing the later software production phases to focus on more complex, functional, and algorithmic faults. A majority of the defects found by automated static analysis appear to be produced by a few key types of programmer error and some of these types have the potential to cause security vulnerabilities. Statistical analysis results indicate the number of automated static analysis faults can be effective for identifying problem modules. Our results indicate static analysis tools are complementary to other fault-detection techniques for the economic production of a high-quality software product.
The Effectiveness of Automated Static Analysis Tools for Fault Detection and Refactoring Prediction
2009 International Conference on Software Testing Verification and Validation, 2009
Many automated static analysis (ASA) tools have been developed in recent years for detecting software anomalies. The aim of these tools is to help developers to eliminate software defects at early stages and produce more reliable software at a lower cost. Determining the effectiveness of ASA tools requires empirical evaluation. This study evaluates coding concerns reported by three ASA tools on two open source software (OSS) projects with respect to two types of modifications performed in the studied software CVS repositories: corrections of faults that caused failures, and refactoring modifications. The results show that fewer than 3% of the detected faults correspond to the coding concerns reported by the ASA tools. ASA tools were more effective in identifying refactoring modifications and corresponded to about 71% of them. More than 96% of the coding concerns were false positives that do not relate to any fault or refactoring modification.
Due to the constant evolution of technology, each day brings new programming languages, development paradigms, and ways of evaluating processes. This is no different with source code metrics, where there is always new metric classes. To use a software metric to support decisions, it is necessary to understand how to perform the metric collection, calculation, interpretation, and analysis. The tasks of collecting and calculating source code metrics are most often automated, but how should we monitor them during the software development cycle? Our research aims to assist the software engineer to monitor metrics of vulnerability threats present in the source code through a reference prediction model, considering that real world software have non-functional security requirements, which implies the need to know how to monitor these requirements during the software development cycle. As a first result, this paper presents an empirical study on the evolution of the Linux project. Based on static analysis data, we propose low complexity models to study flaws in the Linux source code. About 391 versions of the project were analyzed by mining the official Linux repository using an approach that can be reproduced to perform similar studies. Our results show that it is possible to predict the number of warnings triggered by a static analyzer for a given software project revision as long as the software is continuously monitored.
Analysing Bug Prediction Capabilities of Static Code Metrics in Open Source Software
Lecture Notes in Computer Science, 2008
Open Source Softwares provide a rich resource of empirical research in software engineering. Static code metrics are a good indicator of software quality and maintainability. In this work we have tried to answer the question whether bug predictors obtained from one project can be applied to a different project with reasonable accuracy. Two open source projects Firefox and Apache HTTP Server (AHS) are used for this study. Static code metrics are calculated for both projects using inhouse software and the bug information is obtained from bug databases of these projects. The source code files are classified as clean or buggy using the Decision tree classifier. The classifier is trained on metrics and bug data of Firefox and tested on Apache HTTP Server and vice versa. The results obtained vary with different releases of these projects and can be as good as 92 % of the files correctly classified and as poor as 68 % of the files correctly classified by the trained classifier.