Conducting on-line surveys in software engineering (original) (raw)
Related papers
Survey Guidelines in Software Engineering: An Annotated Review
Background: Survey is a method of research aiming to gather data from a large population of interest. Despite being extensively used in software engineering, survey-based research faces several challenges, such as selecting a representative population sample and designing the data collection instruments.
Addressing the challenges of replications of surveys in software engineering research
2005
Surveys are a popular research tool often used in empirical software engineering studies. While researchers are urged to replicate existing surveys, such replication brings with it challenges. This paper presents a concrete example of a replication of a survey used to determine the extent of adoption of software development best practice.
Investigating probabilistic sampling approaches for large-scale surveys in software engineering
Journal of Software Engineering Research and Development, 2015
Background: Establishing representative samples for Software Engineering surveys is still considered a challenge. Specialized literature often presents limitations on interpreting surveys' results, mainly due to the use of sampling frames established by convenience and non-probabilistic criteria for sampling from them. In this sense, we argue that a strategy to support the systematic establishment of sampling frames from an adequate source of sampling can contribute to improve this scenario. Method: A conceptual framework for supporting large scale sampling in Software Engineering surveys has been organized after performing a set of experiences on designing such strategies and gathering evidence regarding their benefits. The use of this conceptual framework based on a sampling strategy developed for supporting the replication of a survey on characteristics of agility and agile practices in software processes is depicted in this paper. Result: A professional social network (Linkedln) was established as the source of sampling and its groups of interest as the units for searching members to be recruited. It allowed to deal with a sampling frame composed by more than 110,000 members (prospective subjects) distributed over 19 groups of interest. Then, through the similarity levels observed among these groups, eight strata were organized and 7745 members were invited, from which 291 have confirmed participation and answered the questionnaire. Conclusion: The heterogeneity and number of participants in this replication contributed to improve the strength of original survey's results. Therefore, we believe the sharing of this experience, the instruments and plan can be helpful for those researchers and practitioners interested on executing large scale surveys in Software Engineering.
Characterizing Sampling Frames in Software Engineering Surveys
2015
Questionnaire-based Survey is a study strategy commonly applied in Software Engineering. It allows the researchers to perform descriptive largescale investigations without the rigorous control level required by experiments. A critical issue on planning surveys concerns with the characterization of adequate sampling frames and their units of analysis. Therefore, this paper presents the results of a structured review in order to identify how sampling frames and units of analysis have been usually characterized in Software Engineering surveys. This investigation allowed to observe the predominant behavior of sampling by convenience whitin units of analysis composed by individuals retrieved from non-representative sources of sampling. Besides, it was also identified many other design alternatives. Based on these results , a set of recommendations on characterizing sampling frames for software engineering surveys, including the attributes gathered from each kind of unit of analysis (orga...
Lessons Learnt in Conducting Survey Research
2017 IEEE/ACM 5th International Workshop on Conducting Empirical Studies in Industry (CESI)
Surveys constitute an valuable tool to capture a large-scale snapshot of the state of the practice. Apparently trivial to adopt, surveys hide, however, several pitfalls that might hinder rendering the result valid and, thus, useful. Goal: We aim at providing an overview of main pitfalls in software engineering surveys and report on practical ways to deal with them. Method: We build on the experiences we collected in conducting many studies and distill the main lessons learnt. Results: The eight lessons learnt we report cover different aspects of the survey process ranging from the design of initial research objectives to the design of a questionnaire. Conclusions: Our hope is that by sharing our lessons learnt, combined with a disciplined application of the general survey theory, we contribute to improving the quality of the research results achievable by employing software engineering surveys.
Contemporary Empirical Methods in Software Engineering, 2020
While being an important and often used research method, survey research has been less often discussed on a methodological level in empirical software engineering than other types of research. This chapter compiles a set of important and challenging issues in survey research based on experiences with several large-scale international surveys. The chapter covers theory building, sampling, invitation and follow-up, statistical as well as qualitative analysis of survey data and the usage of psychometrics in software engineering surveys.
Support Mechanisms to Conduct Empirical Studies in Software Engineering
Context: Empirical studies are gaining recognition in the Software Engineering(SE) research community. In order to foster empirical research, it is essential understand the environments, guidelines, process, and other mechanisms available to support these studies in SE. Goal: Identifying the mechanisms used to support the empirical strategies adopted by the researches in the major Empirical Software Engineering (ESE) scientific venues. Method: We performed a systematic mapping study that included all full papers published at EASE, ESEM and ESEJ since their first editions. A total of 898 studies were selected. Results: We provide the full list of identified support mechanisms and the strategies that uses them. The most commonly mechanisms used to support the empirical strategies were two sets of guidelines, one to secondary studies and another to experiments. The most reported empirical strategies are experiments and case studies. Conclusions: The use of empirical methods in SE has increased over the years but many studies do not apply these methodsnor use mechanisms to guide their research. Therefore, the list of support mechanisms, where and how they were applied is a major asset to the SE community. Such asset can foster empirical studies aiding the choice regarding which strategies and mechanisms to use in a research. Also, we identified new perspectives and gaps that foster the development of resources to aid empirical studies.
ACM SIGSOFT Software Engineering Notes, 2002
This article is the fifth installment of our series of articles on survey research. In it, we discuss what we mean by a population and a sample and the implications of each for survey research. We provide examples of correct and incorrect sampling techniques used in software engineering surveys.
Status of Empirical Research in Software Engineering
Lecture Notes in Computer Science, 2007
We provide an assessment of the status of empirical software research by analyzing all refereed articles that appeared in the Journal of Empirical Software Engineering from its first issue in January 1996 through June 2006. The journal publishes empirical software research exclusively and it is the only journal to do so. The main findings are: 1. The dominant empirical methods are experiments and case studies. Other methods (correlational studies, meta analysis, surveys, descriptive approaches, ex post facto studies) occur infrequently; long-term studies are missing. About a quarter of the experiments are replications. 2. Professionals are used somewhat more frequently than students as subjects. 3. The dominant topics studied are measurement/metrics and tools/methods/frameworks. Metrics research is dominated by correlational and case studies without any experiments. 4. Important topics are underrepresented or absent, for example: programming languages, model driven development, formal methods, and others. The narrow focus on a few empirically researched topics is in contrast to the broad scope of software research.