Challenges in Survey Research (original) (raw)
Related papers
Lessons Learnt in Conducting Survey Research
2017 IEEE/ACM 5th International Workshop on Conducting Empirical Studies in Industry (CESI)
Surveys constitute an valuable tool to capture a large-scale snapshot of the state of the practice. Apparently trivial to adopt, surveys hide, however, several pitfalls that might hinder rendering the result valid and, thus, useful. Goal: We aim at providing an overview of main pitfalls in software engineering surveys and report on practical ways to deal with them. Method: We build on the experiences we collected in conducting many studies and distill the main lessons learnt. Results: The eight lessons learnt we report cover different aspects of the survey process ranging from the design of initial research objectives to the design of a questionnaire. Conclusions: Our hope is that by sharing our lessons learnt, combined with a disciplined application of the general survey theory, we contribute to improving the quality of the research results achievable by employing software engineering surveys.
Addressing the challenges of replications of surveys in software engineering research
2005
Surveys are a popular research tool often used in empirical software engineering studies. While researchers are urged to replicate existing surveys, such replication brings with it challenges. This paper presents a concrete example of a replication of a survey used to determine the extent of adoption of software development best practice.
ACM SIGSOFT Software Engineering Notes, 2002
This article is the fifth installment of our series of articles on survey research. In it, we discuss what we mean by a population and a sample and the implications of each for survey research. We provide examples of correct and incorrect sampling techniques used in software engineering surveys.
Survey Guidelines in Software Engineering: An Annotated Review
Background: Survey is a method of research aiming to gather data from a large population of interest. Despite being extensively used in software engineering, survey-based research faces several challenges, such as selecting a representative population sample and designing the data collection instruments.
Conducting on-line surveys in software engineering
2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings., 2003
One purpose of empirical software engineering is to enable an understanding of factors that influence software development. Surveys are an appropriate empirical strategy to gather data from a large population (e.g., about methods, tools, developers, companies) and to achieve an understanding of that population. Although surveys are quite often performed, for example, in social sciences and marketing research, they are underrepresented in empirical software engineering research, which most often uses controlled experiments and case studies. Consequently, also the methodological support how to perform such studies in software engineering, is rather low. However, with the increasing pervasion of the internet it is possible to perform surveys easily and cost-effectively over internet pages (i.e., online), while at the same time the interest in performing surveys is growing.
Principles of survey research: part 5: populations and samples
ACM SIGSOFT Software Engineering Notes, 2002
This article is the fifth installment of our series of articles on survey research. In it, we discuss what we mean by a population and a sample and the implications of each for survey research. We provide examples of correct and incorrect sampling techniques used in software engineering surveys.
Characterizing Sampling Frames in Software Engineering Surveys
2015
Questionnaire-based Survey is a study strategy commonly applied in Software Engineering. It allows the researchers to perform descriptive largescale investigations without the rigorous control level required by experiments. A critical issue on planning surveys concerns with the characterization of adequate sampling frames and their units of analysis. Therefore, this paper presents the results of a structured review in order to identify how sampling frames and units of analysis have been usually characterized in Software Engineering surveys. This investigation allowed to observe the predominant behavior of sampling by convenience whitin units of analysis composed by individuals retrieved from non-representative sources of sampling. Besides, it was also identified many other design alternatives. Based on these results , a set of recommendations on characterizing sampling frames for software engineering surveys, including the attributes gathered from each kind of unit of analysis (orga...
Principles of Survey Research Part 1" Turning Lemons into Lemonade Setting the scene
Surveys are probably the most commonly-used research method world-wide. Survey work is visible not only because we see many examples of it in software engineering research, but also because we are often asked to participate in surveys in our private capacity, as electors, consumers, or service users. This widespread use of surveys may give us the impression that surveybased research is straightforward, an easy option for researchers to gather important information about products, context, processes, workers and more. In our personal experience with applying and evaluating research methods and their results, we certainly did not expect to encounter major problems with a survey that we planned, to investigate issues associated with technology adoption. This article and subsequent ones in this series describe how wrong we were. We do not want to give the impression that there is any way of turning a bad survey into a good one; if a survey is a lemon, it stays a lemon. However, we believe that learning from our mistakes is the way to make lemonade from lemons. So this series of articles shares with you our lessons learned, in the hope of improving survey research in software engineering.
Investigating probabilistic sampling approaches for large-scale surveys in software engineering
Journal of Software Engineering Research and Development, 2015
Background: Establishing representative samples for Software Engineering surveys is still considered a challenge. Specialized literature often presents limitations on interpreting surveys' results, mainly due to the use of sampling frames established by convenience and non-probabilistic criteria for sampling from them. In this sense, we argue that a strategy to support the systematic establishment of sampling frames from an adequate source of sampling can contribute to improve this scenario. Method: A conceptual framework for supporting large scale sampling in Software Engineering surveys has been organized after performing a set of experiences on designing such strategies and gathering evidence regarding their benefits. The use of this conceptual framework based on a sampling strategy developed for supporting the replication of a survey on characteristics of agility and agile practices in software processes is depicted in this paper. Result: A professional social network (Linkedln) was established as the source of sampling and its groups of interest as the units for searching members to be recruited. It allowed to deal with a sampling frame composed by more than 110,000 members (prospective subjects) distributed over 19 groups of interest. Then, through the similarity levels observed among these groups, eight strata were organized and 7745 members were invited, from which 291 have confirmed participation and answered the questionnaire. Conclusion: The heterogeneity and number of participants in this replication contributed to improve the strength of original survey's results. Therefore, we believe the sharing of this experience, the instruments and plan can be helpful for those researchers and practitioners interested on executing large scale surveys in Software Engineering.