Tore Dybå | University of Oslo (original) (raw)

Papers by Tore Dybå

IEEE Transactions on Software Engineering

Context: Code smells are assumed to indicate bad design that leads to less maintainable code. How... more Context: Code smells are assumed to indicate bad design that leads to less maintainable code. However, this assumption has not been investigated in controlled studies with professional software developers. Aim: This paper investigates the relationship between code smells and maintenance effort. Method: Six developers were hired to perform three maintenance tasks each on four functionally equivalent Java systems originally implemented by different companies. Each developer spent three to four weeks. In total, they modified 298 Java files in the four systems. An Eclipse IDE plug-in measured the exact amount of time a developer spent maintaining each file. Regression analysis was used to explain the effort using file properties, including the number of smells. Results: None of the 12 investigated smells was significantly associated with increased effort after we adjusted for file size and the number of changes; Refused Bequest was significantly associated with decreased effort. File size and the number of changes explained almost all of the modeled variation in effort. Conclusion: The effects of the 12 smells on maintenance effort were limited. To reduce maintenance effort, a focus on reducing code size and the work practices that limit the number of changes may be more beneficial than refactoring code smells.

Lecture Notes in Business Information Processing, 2012

Escalating commitment is a common and costly phenomenon in software projects in which decision-ma... more Escalating commitment is a common and costly phenomenon in software projects in which decision-makers continue to invest resources to a failing course of action. We conducted a longitudinal case study exploring the effect of daily meetings on escalating commitment. This was done in an agile project building software for the oil and gas industry. By analyzing data collected over a period of four years, and drawing on concepts from selfjustification theory we found that daily meetings contributed to maintain a situation of escalating commitment. This especially occurs if the meeting becomes a place for reporting and defending decisions with team members feeling that they have to justify their choices towards the project management or fellow team members. Early signs of escalation such as rationalizing continuation of a chosen course of action must therefore be taken seriously.

Lecture Notes in Computer Science, 2004

... 179-211. 2. Baruch, Y. (1999) Response Rate in Academic Studies - A Comparative Analysis, Hum... more ... 179-211. 2. Baruch, Y. (1999) Response Rate in Academic Studies - A Comparative Analysis, Human Relations, vol. 52, no. ... 318-339. 4. Fichman, RG and Kemmerer, CF (1993) Adoption of Software Engineering Process In-novations, Sloan Management Review, vol. 34, no. ...

Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement - ESEM '10, 2010

Although Scrum is an important topic in software engineering and information systems, few longitu... more Although Scrum is an important topic in software engineering and information systems, few longitudinal industrial studies have investigated the effects of Scrum on software quality, in terms of defects and defect density, and the quality assurance process. In this paper we report on a longitudinal study in which we have followed a project over a three-year period. We compared software quality assurance processes and software defects of the project between a 17-month phase with a plan-driven process, followed by a 20-month phase with Scrum. The results of the study did not show a significant reduction of defect densities or changes of defect profiles after Scrum was used. However, the iterative nature of Scrum resulted in constant system and acceptance testing and related defect fixing, which made the development process more efficient in terms of fewer surprises and better control of software quality and release date. In addition, software quality and knowledge sharing got more focus when using Scrum. However, Scrum put more stress and time pressure on the developers, and made them reluctant to perform certain tasks for later maintenance, such as refactoring.

First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007), 2007

Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement - ESEM '08, 2008

Systematic reviews are only as good as the evidence they are based on. It is important, therefore... more Systematic reviews are only as good as the evidence they are based on. It is important, therefore, that users of systematic reviews know how much confidence they can place in the conclusions and recommendations arising from such reviews. In this paper we present an overview of some of the most influential systems for assessing the quality of individual primary studies and for grading the overall strength of a body of evidence. We also present an example of the use of such systems based on a systematic review of empirical studies of agile software development. Our findings suggest that the systems used in other disciplines for grading the strength of evidence for and reporting of systematic reviews, especially those that take account of qualitative and observational studies are of particular relevance for software engineering.

2012 5th International Workshop on Co-operative and Human Aspects of Software Engineering (CHASE), 2012

Software development is most often done in teams, where human and cooperative aspects are vital f... more Software development is most often done in teams, where human and cooperative aspects are vital for team effectiveness. This has been the topic of study in several disciplines, and in this article we describe three team effectiveness models from other fields. We discuss priorities for future studies on software teams, and ask: Do we need our own effectiveness model for software teams?

Faster, better and cheaper are challenges that IT-companies face every day. The customer's ex... more Faster, better and cheaper are challenges that IT-companies face every day. The customer's expectations shall be met in a world where constant change in environment, organization and technology are the rule rather that the exception. A solution for meeting these challenges is to share knowledge and experience - use the company's own experience, and the experience of other companies. Process Improvement in Practice - A Handbook for IT Companies tackles the problems involved in launching these solutions. Process Improvement in Practice - A Handbook for IT Companies is designed for small IT companies who wish to start with systematic improvement. The methods and techniques in this handbook are tried in practice, and have proven to be easy to use and scalable for local needs. Managers and developers will discover useful tips to initiate improvement work efficiently. This practical handbook is based on the authors' improvement work in a range of companies since the mid-nineti...

IEEE Transactions on Software Engineering, 2014

Skilled workers are crucial to the success of software development. The current practice in resea... more Skilled workers are crucial to the success of software development. The current practice in research and industry for assessing programming skills is mostly to use proxy variables of skill, such as education, experience, and multiple-choice knowledge tests. There is as yet no valid and efficient way to measure programming skill. The aim of this research is to develop a valid instrument that measures programming skill by inferring skill directly from the performance on programming tasks. Over two days, 65 professional developers from eight countries solved 19 Java programming tasks. Based on the developers' performance, the Rasch measurement model was used to construct the instrument. The instrument was found to have satisfactory (internal) psychometric properties and correlated with external variables in compliance with theoretical expectations. Such an instrument has many implications for practice, for example, in job recruitment and project allocation.

Software Process Improvement, 2006

Objective: Our objective is to perform an empirical investigation on factors affecting software d... more Objective: Our objective is to perform an empirical investigation on factors affecting software developer acceptance and utilization of electronic process guides (EPGs) and to discuss the implications of the findings.

Lecture Notes in Computer Science, 2003

Software development is an experimental discipline, i.e. somewhat unpredictable. This suggests th... more Software development is an experimental discipline, i.e. somewhat unpredictable. This suggests that software processes improvement should be based on the continuous iteration of characterization, goal setting, selection of improved technology, monitoring and analysis of its effects. This paper describes experiences from the empirical studies in two large SPI programmes in Norway. Five main lessons were learned: 1) It is a challenge for the industrial partners to invest enough resources in SPI activities. 2) The research partners must learn to know the companies, and 3) they must work as a multicompetent and coherent unit towards them. 4) Any SPI initiative must show visible, short-term payoff. 5) Establishing a solid baseline from which to improve is unrealistic. Based on these lessons, a set of operational recommendations for other researchers in the area are proposed.

International Series in Software Engineering, 2005

We introduce electronic process guides, and discuss their role in software engineering projects. ... more We introduce electronic process guides, and discuss their role in software engineering projects. We then present existing methods for constructing electronic process guides by defining a set of common processes for a company. Different approaches from the software engineering and management science are presented. We then go on to propose a new way of dealing with process description in software engineering: using process workshops as a tool to reach consensus on work practice. The main reason for this is to get realistic descriptions with accurate detail as well as company commitment in an efficient manner. We describe our workshop-oriented method to define processes, which we have used in small software companies, and show examples of results.

ACM SIGSOFT Software Engineering Notes, 2001

Most quality and software process improvement frameworks emphasize written (i.e. formal) document... more Most quality and software process improvement frameworks emphasize written (i.e. formal) documentation to convey recommended work practices. However, there is considerable skepticism among developers to learn from and adhere to prescribed process models. The latter are often perceived as overly "structured" or implying too much "control". Further, what is relevant knowledge has often been decided by "others"---often the quality manager.

Lecture Notes in Computer Science, 2003

An important goal of most empirical software engineering experiments is the transfer of the resea... more An important goal of most empirical software engineering experiments is the transfer of the research results to industrial applications. To convince industry about the validity and applicability of the results of controlled software engineering experiments, the tasks, subjects and the environments should be as realistic as practically possible. Such experiments are, however, more demanding and expensive than experiments involving students, small tasks and pen-and-paper environments. This chapter describes challenges of increasing the realism of controlled experiments and lessons learned from the experiments that have been conducted at Simula Research Laboratory.

2011 International Symposium on Empirical Software Engineering and Measurement, 2011

The skills of software developers are crucial to the success of software projects. Also, controll... more The skills of software developers are crucial to the success of software projects. Also, controlling for individual differences is important when studying the general effect of a method or a tool. However, the way skill is determined in industry and research settings is often ad hoc or based on unvalidated methods. According to established test theory, validated tests of skill should infer skill levels from welldefined performance measures on multiple small, representative tasks. We show how time and quality can be meaningfully combined to a well-defined measure of small-task performance, and hence a measure of programming skill. Our results show significant and positive correlations between our proposed measures of skill and variables such as seniority or self-evaluated expertise. These methods for combining time and quality are a promising first step to measuring programming skill in industry and research settings.

For process models to be useful, more and more software companies not only tailor their process m... more For process models to be useful, more and more software companies not only tailor their process models to the specific needs of the company, but also make them available on the company's intranet. In this article, we try to understand what characterizes the use of an electronic process guide (EPG) among developers in a medium-sized software company. We describe the findings that emerged from a grounded theory study in which we interviewed 19 developers and project managers. We found that implementing an EPG is not a straightforward process, but that it depends on several organizational factors. Our findings suggest that the process guide should be anchored among the leaders, that users should be involved in creating and updating it and offered training and support during use, that efforts should be taken to keep the guide updated, that templates and checklists are seen as the most useful features, that integration with other tools make it more useful, and, finally, that it should be easy to tailor the process guide to different project types or activities. The insights gained from this study can be used as recommendations for other medium-sized software companies when implementing electronic process guides.

Background: Experiments in which study units are assigned to experimental groups nonrandomly are ... more Background: Experiments in which study units are assigned to experimental groups nonrandomly are called quasi-experiments. They allow investigations of cause-effect relations in settings in which randomization is inappropriate, impractical, or too costly. Problem outline: The procedure by which the nonrandom assignments are made might result in selection bias and other related internal validity problems. Selection bias is a systematic (not happening by chance) pre-experimental difference between the groups that could influence the results. By detecting the cause of the selection bias, and designing and analyzing the experiments accordingly, the effect of the bias may be reduced or eliminated. Research method: To investigate how quasi-experiments are performed in software engineering (SE), we conducted a systematic review of the experiments published in nine major SE journals and three conference proceedings in the decade 1993-2002. Results: Among the 113 experiments detected, 35% were quasi-experiments. In addition to field experiments, we found several applications for quasi-experiments in SE. However, there seems to be little awareness of the precise nature of quasi-experiments and the potential for selection bias in them. The term ''quasi-experiment" was used in only 10% of the articles reporting quasi-experiments; only half of the quasi-experiments measured a pretest score to control for selection bias, and only 8% reported a threat of selection bias. On average, larger effect sizes were seen in randomized than in quasi-experiments, which might be due to selection bias in the quasi-experiments. Conclusion: We conclude that quasi-experimentation is useful in many settings in SE, but their design and analysis must be improved (in ways described in this paper), to ensure that inferences made from this kind of experiment are valid.

Empirical Software Engineering, 2014

Case studies are largely used for investigating software engineering practices. They are characte... more Case studies are largely used for investigating software engineering practices. They are characterized by their flexible nature, multiple forms of data collection, and are mostly informed by qualitative data. Synthesis of case studies is necessary to build a body of knowledge from individual cases. There are many methods for such synthesis, but they are yet not well explored in software engineering. The objective of this research is to demonstrate the similarities and differences of the results and conclusions when applying three different methods of synthesis, and to discuss the challenges of synthesizing evidence from reported case studies in SE. We describe a worked example of three such methods where three independent teams synthesized two studies that investigated critical factors of trust in outsourced projects through thematic synthesis and cross-case analysis, and compared these to each other and also to an already published narrative synthesis. In addition, despite that the primary studies were well presented for synthesis, we identified challenges in the use of case studies synthesis methods related to the goals and research questions of the synthesis, the types and number of case studies, variations in context, limited access to raw data, and quality of the case studies. Thus, we recommend that the analysts should be aware of these challenges and try to account for them during the execution of the synthesis. We also Empir Software Eng recommend that analysts consider using more than one method of synthesis for sake of reliability of the results and conclusions.

2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings., 2003

In recent years, several claims have been put forward in favour of pair programming, as opposed t... more In recent years, several claims have been put forward in favour of pair programming, as opposed to individual programming. However, results from existing studies on pair programming contain apparent contradictions. The differences in the context in which the studies were conducted may be one explanation for such results. This paper presents an initial framework for research on pair programming. The aim is to support empirical studies and meta-analysis for developing theories about pair programming. The framework is based on (1) existing studies on pair programming, (2) ongoing studies by the authors, and (3) theories from group dynamics.

Future of Software Engineering (FOSE '07), 2007

We present the vision that for all fields of software engineering (SE), empirical research method... more We present the vision that for all fields of software engineering (SE), empirical research methods should enable the development of scientific knowledge about how useful different SE technologies are for different kinds of actors, performing different kinds of activities, on different kinds of systems. It is part of the vision that such scientific knowledge will guide the development of new SE technology and is a major input to important SE decisions in industry. Major challenges to the pursuit of this vision are: more SE research should be based on the use of empirical methods; the quality, including relevance, of the studies using such methods should be increased; there should be more and better synthesis of empirical evidence; and more theories should be built and tested. Means to meet these challenges include (1) increased competence regarding how to apply and combine alternative empirical methods, (2) tighter links between academia and industry, (3) the development of common research agendas with a focus on empirical methods, and (4) more resources for empirical research. Future of Software Engineering(FOSE'07) 0-7695-2829-5/07 $20.00

IEEE Transactions on Software Engineering

Context: Code smells are assumed to indicate bad design that leads to less maintainable code. How... more Context: Code smells are assumed to indicate bad design that leads to less maintainable code. However, this assumption has not been investigated in controlled studies with professional software developers. Aim: This paper investigates the relationship between code smells and maintenance effort. Method: Six developers were hired to perform three maintenance tasks each on four functionally equivalent Java systems originally implemented by different companies. Each developer spent three to four weeks. In total, they modified 298 Java files in the four systems. An Eclipse IDE plug-in measured the exact amount of time a developer spent maintaining each file. Regression analysis was used to explain the effort using file properties, including the number of smells. Results: None of the 12 investigated smells was significantly associated with increased effort after we adjusted for file size and the number of changes; Refused Bequest was significantly associated with decreased effort. File size and the number of changes explained almost all of the modeled variation in effort. Conclusion: The effects of the 12 smells on maintenance effort were limited. To reduce maintenance effort, a focus on reducing code size and the work practices that limit the number of changes may be more beneficial than refactoring code smells.

Lecture Notes in Business Information Processing, 2012

Escalating commitment is a common and costly phenomenon in software projects in which decision-ma... more Escalating commitment is a common and costly phenomenon in software projects in which decision-makers continue to invest resources to a failing course of action. We conducted a longitudinal case study exploring the effect of daily meetings on escalating commitment. This was done in an agile project building software for the oil and gas industry. By analyzing data collected over a period of four years, and drawing on concepts from selfjustification theory we found that daily meetings contributed to maintain a situation of escalating commitment. This especially occurs if the meeting becomes a place for reporting and defending decisions with team members feeling that they have to justify their choices towards the project management or fellow team members. Early signs of escalation such as rationalizing continuation of a chosen course of action must therefore be taken seriously.

Lecture Notes in Computer Science, 2004

... 179-211. 2. Baruch, Y. (1999) Response Rate in Academic Studies - A Comparative Analysis, Hum... more ... 179-211. 2. Baruch, Y. (1999) Response Rate in Academic Studies - A Comparative Analysis, Human Relations, vol. 52, no. ... 318-339. 4. Fichman, RG and Kemmerer, CF (1993) Adoption of Software Engineering Process In-novations, Sloan Management Review, vol. 34, no. ...

Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement - ESEM '10, 2010

Although Scrum is an important topic in software engineering and information systems, few longitu... more Although Scrum is an important topic in software engineering and information systems, few longitudinal industrial studies have investigated the effects of Scrum on software quality, in terms of defects and defect density, and the quality assurance process. In this paper we report on a longitudinal study in which we have followed a project over a three-year period. We compared software quality assurance processes and software defects of the project between a 17-month phase with a plan-driven process, followed by a 20-month phase with Scrum. The results of the study did not show a significant reduction of defect densities or changes of defect profiles after Scrum was used. However, the iterative nature of Scrum resulted in constant system and acceptance testing and related defect fixing, which made the development process more efficient in terms of fewer surprises and better control of software quality and release date. In addition, software quality and knowledge sharing got more focus when using Scrum. However, Scrum put more stress and time pressure on the developers, and made them reluctant to perform certain tasks for later maintenance, such as refactoring.

First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007), 2007

Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement - ESEM '08, 2008

Systematic reviews are only as good as the evidence they are based on. It is important, therefore... more Systematic reviews are only as good as the evidence they are based on. It is important, therefore, that users of systematic reviews know how much confidence they can place in the conclusions and recommendations arising from such reviews. In this paper we present an overview of some of the most influential systems for assessing the quality of individual primary studies and for grading the overall strength of a body of evidence. We also present an example of the use of such systems based on a systematic review of empirical studies of agile software development. Our findings suggest that the systems used in other disciplines for grading the strength of evidence for and reporting of systematic reviews, especially those that take account of qualitative and observational studies are of particular relevance for software engineering.

2012 5th International Workshop on Co-operative and Human Aspects of Software Engineering (CHASE), 2012

Software development is most often done in teams, where human and cooperative aspects are vital f... more Software development is most often done in teams, where human and cooperative aspects are vital for team effectiveness. This has been the topic of study in several disciplines, and in this article we describe three team effectiveness models from other fields. We discuss priorities for future studies on software teams, and ask: Do we need our own effectiveness model for software teams?

Faster, better and cheaper are challenges that IT-companies face every day. The customer's ex... more Faster, better and cheaper are challenges that IT-companies face every day. The customer's expectations shall be met in a world where constant change in environment, organization and technology are the rule rather that the exception. A solution for meeting these challenges is to share knowledge and experience - use the company's own experience, and the experience of other companies. Process Improvement in Practice - A Handbook for IT Companies tackles the problems involved in launching these solutions. Process Improvement in Practice - A Handbook for IT Companies is designed for small IT companies who wish to start with systematic improvement. The methods and techniques in this handbook are tried in practice, and have proven to be easy to use and scalable for local needs. Managers and developers will discover useful tips to initiate improvement work efficiently. This practical handbook is based on the authors' improvement work in a range of companies since the mid-nineti...

IEEE Transactions on Software Engineering, 2014

Skilled workers are crucial to the success of software development. The current practice in resea... more Skilled workers are crucial to the success of software development. The current practice in research and industry for assessing programming skills is mostly to use proxy variables of skill, such as education, experience, and multiple-choice knowledge tests. There is as yet no valid and efficient way to measure programming skill. The aim of this research is to develop a valid instrument that measures programming skill by inferring skill directly from the performance on programming tasks. Over two days, 65 professional developers from eight countries solved 19 Java programming tasks. Based on the developers' performance, the Rasch measurement model was used to construct the instrument. The instrument was found to have satisfactory (internal) psychometric properties and correlated with external variables in compliance with theoretical expectations. Such an instrument has many implications for practice, for example, in job recruitment and project allocation.

Software Process Improvement, 2006

Objective: Our objective is to perform an empirical investigation on factors affecting software d... more Objective: Our objective is to perform an empirical investigation on factors affecting software developer acceptance and utilization of electronic process guides (EPGs) and to discuss the implications of the findings.

Lecture Notes in Computer Science, 2003

Software development is an experimental discipline, i.e. somewhat unpredictable. This suggests th... more Software development is an experimental discipline, i.e. somewhat unpredictable. This suggests that software processes improvement should be based on the continuous iteration of characterization, goal setting, selection of improved technology, monitoring and analysis of its effects. This paper describes experiences from the empirical studies in two large SPI programmes in Norway. Five main lessons were learned: 1) It is a challenge for the industrial partners to invest enough resources in SPI activities. 2) The research partners must learn to know the companies, and 3) they must work as a multicompetent and coherent unit towards them. 4) Any SPI initiative must show visible, short-term payoff. 5) Establishing a solid baseline from which to improve is unrealistic. Based on these lessons, a set of operational recommendations for other researchers in the area are proposed.

International Series in Software Engineering, 2005

We introduce electronic process guides, and discuss their role in software engineering projects. ... more We introduce electronic process guides, and discuss their role in software engineering projects. We then present existing methods for constructing electronic process guides by defining a set of common processes for a company. Different approaches from the software engineering and management science are presented. We then go on to propose a new way of dealing with process description in software engineering: using process workshops as a tool to reach consensus on work practice. The main reason for this is to get realistic descriptions with accurate detail as well as company commitment in an efficient manner. We describe our workshop-oriented method to define processes, which we have used in small software companies, and show examples of results.

ACM SIGSOFT Software Engineering Notes, 2001

Most quality and software process improvement frameworks emphasize written (i.e. formal) document... more Most quality and software process improvement frameworks emphasize written (i.e. formal) documentation to convey recommended work practices. However, there is considerable skepticism among developers to learn from and adhere to prescribed process models. The latter are often perceived as overly "structured" or implying too much "control". Further, what is relevant knowledge has often been decided by "others"---often the quality manager.

Lecture Notes in Computer Science, 2003

An important goal of most empirical software engineering experiments is the transfer of the resea... more An important goal of most empirical software engineering experiments is the transfer of the research results to industrial applications. To convince industry about the validity and applicability of the results of controlled software engineering experiments, the tasks, subjects and the environments should be as realistic as practically possible. Such experiments are, however, more demanding and expensive than experiments involving students, small tasks and pen-and-paper environments. This chapter describes challenges of increasing the realism of controlled experiments and lessons learned from the experiments that have been conducted at Simula Research Laboratory.

2011 International Symposium on Empirical Software Engineering and Measurement, 2011

The skills of software developers are crucial to the success of software projects. Also, controll... more The skills of software developers are crucial to the success of software projects. Also, controlling for individual differences is important when studying the general effect of a method or a tool. However, the way skill is determined in industry and research settings is often ad hoc or based on unvalidated methods. According to established test theory, validated tests of skill should infer skill levels from welldefined performance measures on multiple small, representative tasks. We show how time and quality can be meaningfully combined to a well-defined measure of small-task performance, and hence a measure of programming skill. Our results show significant and positive correlations between our proposed measures of skill and variables such as seniority or self-evaluated expertise. These methods for combining time and quality are a promising first step to measuring programming skill in industry and research settings.

For process models to be useful, more and more software companies not only tailor their process m... more For process models to be useful, more and more software companies not only tailor their process models to the specific needs of the company, but also make them available on the company's intranet. In this article, we try to understand what characterizes the use of an electronic process guide (EPG) among developers in a medium-sized software company. We describe the findings that emerged from a grounded theory study in which we interviewed 19 developers and project managers. We found that implementing an EPG is not a straightforward process, but that it depends on several organizational factors. Our findings suggest that the process guide should be anchored among the leaders, that users should be involved in creating and updating it and offered training and support during use, that efforts should be taken to keep the guide updated, that templates and checklists are seen as the most useful features, that integration with other tools make it more useful, and, finally, that it should be easy to tailor the process guide to different project types or activities. The insights gained from this study can be used as recommendations for other medium-sized software companies when implementing electronic process guides.

Background: Experiments in which study units are assigned to experimental groups nonrandomly are ... more Background: Experiments in which study units are assigned to experimental groups nonrandomly are called quasi-experiments. They allow investigations of cause-effect relations in settings in which randomization is inappropriate, impractical, or too costly. Problem outline: The procedure by which the nonrandom assignments are made might result in selection bias and other related internal validity problems. Selection bias is a systematic (not happening by chance) pre-experimental difference between the groups that could influence the results. By detecting the cause of the selection bias, and designing and analyzing the experiments accordingly, the effect of the bias may be reduced or eliminated. Research method: To investigate how quasi-experiments are performed in software engineering (SE), we conducted a systematic review of the experiments published in nine major SE journals and three conference proceedings in the decade 1993-2002. Results: Among the 113 experiments detected, 35% were quasi-experiments. In addition to field experiments, we found several applications for quasi-experiments in SE. However, there seems to be little awareness of the precise nature of quasi-experiments and the potential for selection bias in them. The term ''quasi-experiment" was used in only 10% of the articles reporting quasi-experiments; only half of the quasi-experiments measured a pretest score to control for selection bias, and only 8% reported a threat of selection bias. On average, larger effect sizes were seen in randomized than in quasi-experiments, which might be due to selection bias in the quasi-experiments. Conclusion: We conclude that quasi-experimentation is useful in many settings in SE, but their design and analysis must be improved (in ways described in this paper), to ensure that inferences made from this kind of experiment are valid.

Empirical Software Engineering, 2014

Case studies are largely used for investigating software engineering practices. They are characte... more Case studies are largely used for investigating software engineering practices. They are characterized by their flexible nature, multiple forms of data collection, and are mostly informed by qualitative data. Synthesis of case studies is necessary to build a body of knowledge from individual cases. There are many methods for such synthesis, but they are yet not well explored in software engineering. The objective of this research is to demonstrate the similarities and differences of the results and conclusions when applying three different methods of synthesis, and to discuss the challenges of synthesizing evidence from reported case studies in SE. We describe a worked example of three such methods where three independent teams synthesized two studies that investigated critical factors of trust in outsourced projects through thematic synthesis and cross-case analysis, and compared these to each other and also to an already published narrative synthesis. In addition, despite that the primary studies were well presented for synthesis, we identified challenges in the use of case studies synthesis methods related to the goals and research questions of the synthesis, the types and number of case studies, variations in context, limited access to raw data, and quality of the case studies. Thus, we recommend that the analysts should be aware of these challenges and try to account for them during the execution of the synthesis. We also Empir Software Eng recommend that analysts consider using more than one method of synthesis for sake of reliability of the results and conclusions.

2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings., 2003

In recent years, several claims have been put forward in favour of pair programming, as opposed t... more In recent years, several claims have been put forward in favour of pair programming, as opposed to individual programming. However, results from existing studies on pair programming contain apparent contradictions. The differences in the context in which the studies were conducted may be one explanation for such results. This paper presents an initial framework for research on pair programming. The aim is to support empirical studies and meta-analysis for developing theories about pair programming. The framework is based on (1) existing studies on pair programming, (2) ongoing studies by the authors, and (3) theories from group dynamics.

Future of Software Engineering (FOSE '07), 2007

We present the vision that for all fields of software engineering (SE), empirical research method... more We present the vision that for all fields of software engineering (SE), empirical research methods should enable the development of scientific knowledge about how useful different SE technologies are for different kinds of actors, performing different kinds of activities, on different kinds of systems. It is part of the vision that such scientific knowledge will guide the development of new SE technology and is a major input to important SE decisions in industry. Major challenges to the pursuit of this vision are: more SE research should be based on the use of empirical methods; the quality, including relevance, of the studies using such methods should be increased; there should be more and better synthesis of empirical evidence; and more theories should be built and tested. Means to meet these challenges include (1) increased competence regarding how to apply and combine alternative empirical methods, (2) tighter links between academia and industry, (3) the development of common research agendas with a focus on empirical methods, and (4) more resources for empirical research. Future of Software Engineering(FOSE'07) 0-7695-2829-5/07 $20.00