Support Mechanisms to Conduct Empirical Studies in Software Engineering (original) (raw)
Related papers
Empirical Strategies in Software Engineering Research: A Literature Survey
2021 Second International Conference on Information Systems and Software Technologies (ICI2ST), 2021
The Software Engineering (SE) research continues to gain strength and interest for researchers considering the need to apply rigor and scientific validity to research results. Objective: Establishing an overview of the topic through a classification scheme of publications and structure the field of interest. Method: We conducted a Systematic Mapping Study, including articles published until 2019, that report at least one study of empirical strategies in SE. Results: 80 initial sets of studies were selected and analyzed, identifying: i) empirical strategy type used and ii) Software Engineering hypotheses types used. Also, 20 papers of the set of studies for mapping were selected and analyzed, identifying 17 empirical strategies and 11 main characteristics to address the empirical research inception in SE. Conclusions: We corroborate that the selection of an empirical strategy in Software Engineering research depends on the nature and scope of the research and on the resources that the researcher has at that moment, in addition to the degree of scientific and methodological knowledge that he has to carry out an empirical study. It is necessary to continue studying in-depth the behavior and nature of the empirical strategies in Software Engineering research that allows strengthening the scientific taxonomy in SE, besides walking towards the automation of the experimental process.
The Future of Empirical Methods in Software Engineering Research
2007
We present the vision that for all fields of software engineering (SE), empirical research methods should enable the development of scientific knowledge about how useful different SE technologies are for different kinds of actors, performing different kinds of activities, on different kinds of systems. It is part of the vision that such scientific knowledge will guide the development of new SE technology and is a major input to important SE decisions in industry. Major challenges to the pursuit of this vision are: more SE research should be based on the use of empirical methods; the quality, including relevance, of the studies using such methods should be increased; there should be more and better synthesis of empirical evidence; and more theories should be built and tested. Means to meet these challenges include (1) increased competence regarding how to apply and combine alternative empirical methods, (2) tighter links between academia and industry, (3) the development of common research agendas with a focus on empirical methods, and (4) more resources for empirical research. Future of Software Engineering(FOSE'07) 0-7695-2829-5/07 $20.00
Status of Empirical Research in Software Engineering
Lecture Notes in Computer Science, 2007
We provide an assessment of the status of empirical software research by analyzing all refereed articles that appeared in the Journal of Empirical Software Engineering from its first issue in January 1996 through June 2006. The journal publishes empirical software research exclusively and it is the only journal to do so. The main findings are: 1. The dominant empirical methods are experiments and case studies. Other methods (correlational studies, meta analysis, surveys, descriptive approaches, ex post facto studies) occur infrequently; long-term studies are missing. About a quarter of the experiments are replications. 2. Professionals are used somewhat more frequently than students as subjects. 3. The dominant topics studied are measurement/metrics and tools/methods/frameworks. Metrics research is dominated by correlational and case studies without any experiments. 4. Important topics are underrepresented or absent, for example: programming languages, model driven development, formal methods, and others. The narrow focus on a few empirically researched topics is in contrast to the broad scope of software research.
2006) ‘Evaluating Guidelines for Empirical Software Engineering
2014
Background. Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Andreas Jedlitschka and Dietmar Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. If guidelines are flawed, they will cause more problems that they solve. Aim. The aim of this paper is to present the method we used to evaluate the guidelines and report the results of our evaluation exercise. We suggest our evaluation process may be of more general use if reporting guidelines for other types of empirical study are developed. Method. We used perspective-based inspections to perform a theoretical evaluation of the guidelines. A separate inspection was performed for each perspective. The perspectives used were:
Directions and methodologies for empirical software engineering research
Empirical Software …, 1999
This report summarises and builds on the results of the "Directions and Methodologies for Empirical Software Engineering Research" group discussion. In particular, we considered the strengths, weaknesses, opportunities and threats to empirical software engineering research in light of the discussions and presentations during the workshop. The following sections describe each of these aspects of our discussion in turn. In addition, to finalise our discussion we agreed on a three-point plan for future work.
IEEE Access
Background: Replication is a recurrent issue in empirical software engineering (ESE). Although it is a foundation of science, replication is hard to execute despite the many supporting tools meant to facilitate reproducibility. For example, in an experiment, which is the most used method in ESE, the number of replications is not enough compared to other sciences. Objective: In this study, we aim to identify tools that maximize reproducibility in software engineering experiments and how they are applied. Methods: We performed a Systematic Mapping Study and complementary strategies to analyze replication from three concerns (communication, knowledge management, and motivation). We analyzed more than 2,600 studies to get 40 primary studies, using a qualitative analytical tool (Atlas.ti) to create semantic maps for synthesizing our results. Result: We found that tools and practices depend on the experiment domain. Human-oriented experiments tend to use an informal mechanism that is costly and time-consuming. On the other hand, technology-oriented experiments are automated, domain-centric, and specialized so they require a learning process and are not transferable to other domains. Conclusion: Tools and practices still lack acceptation and usability among the ESE research community. Therefore, reproducibility is mostly relegated to internal replication, at which time and costs can be assumed within research groups. A focus on new alternatives should be considered to broaden replication.
Evaluating guidelines for reporting empirical software engineering studies
Empirical Software Engineering, 2008
Background Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Jedlitschka and Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. Aim The aim of this
Journal of Systems and Software
Software engineering research is evolving and papers are increasingly based on empirical data from a multitude of sources, using statistical tests to determine if and to what degree empirical evidence supports their hypotheses. To investigate the practices and trends of statistical analysis in empirical software engineering (ESE), this paper presents a review of a large pool of papers from top-ranked software engineering journals. First, we manually reviewed 161 papers and in the second phase of our method, we conducted a more extensive semi-automatic classification of papers spanning the years 2001-2015 and 5,196 papers. Results from both review steps was used to: i) identify and analyse the predominant practices in ESE (e.g., using t-test or ANOVA), as well as relevant trends in usage of specific statistical methods (e.g., nonparametric tests and effect size measures) and, ii) develop a conceptual model for a statistical analysis workflow with suggestions on how to apply different statistical methods as well as guidelines to avoid pitfalls. Lastly, we confirm existing claims that current ESE practices lack a standard to report practical significance of results. We illustrate how practical significance can be discussed in terms of both the statistical analysis and in the practitioner's context.
Mechanisms to Characterize Context of Empirical Studies in Software Engineering
2015
Background: It has become evident that empirical studies in software engineering (SE) have problems related to context characterization. This situation jeopardizes studies replication, result interpretation, knowledge transfer between academia and industry, and evidence integration of secondary studies. Goals: Our goals in this research are to identify and classify the mechanisms that support context characterization of empirical studies in SE. Method: A systematic mapping study with exhaustive coverage was conducted in accordance with the guidelines of evidence-based software engineering. Results: Out of 13,355 studies, 13 studies published between 1999 and 2012 were selected. Only one mechanism adopts the omnibus context approach, against 12 that follow the discrete approach. Ten studies present mechanisms to support context characterization of experiments. Only four out of the ten software engineering topics are covered by the found mechanisms. Conclusions: We found few mechanism...
IEEE Access, 2019
In any discipline, replications of empirical studies are necessary to consolidate the acquired knowledge. In Software Engineering, replications have been reported since the 1990s, although their number is still small. The difficulty in publishing, the lack of guidelines, and the unavailability of replication packages are pointed out by the community as some of the main causes. Objective: Understanding the current state of replications in Software Engineering studies by evaluating current trends and evolution during the last 6 years. Method: A Systematic Mapping Study including articles published in the 2013-2018 period that report at least one replication of an empirical study in Software Engineering. Results: 137 studies were selected and analysed, identifying: i) forums; ii) authors, co-authorships and institutions; iii) most cited studies; iv) research topics addressed; v) empirical methods used; vi) temporal distribution of publications; and vii) distribution of studies according to research topics and empirical methods. Conclusions: According to our results, the most relevant forums are the Empirical Software Engineering and Information and Software Technology journals, and the Empirical Software Engineering and Measurement conference. We observed that, as in previous reviews by other researchers, most of the studies were carried out by European institutions, especially Italian, Spanish, and German researchers and institutions. The studies attracting more citations were published mainly in journals and in the International Conference on Software Engineering. Testing, requirements, and software construction were the most frequent topics of replication studies, whereas the usual empirical method was the controlled experiment. On the other hand, we identified research gaps in areas such as software engineering process, software configuration management, and software engineering economics. When analysed together with previous reviews, there is a clear increasing trend in the number of published replications in the 2013-2018 period. INDEX TERMS Empirical software engineering, replications, systematic mapping study.