Adam Porter | University of Maryland, College Park (original) (raw)

Papers by Adam Porter

[1991 Proceedings] 13th International Conference on Software Engineering

Metric-driven analysis and feedback systems enable developers to define empirically guided proces... more Metric-driven analysis and feedback systems enable developers to define empirically guided processes for software development and maintenance. These systems enable developers to achieve S. E.I. process maturit y levels 4 and 5, "managed" and "optimizing" processes. We define a set of architectural principles and abstract interfaces for designing metric-driven analysis and feedback systems. A fundamental principle is to make measurement active by integrating measurement and process, which contrasts with the primarily passive use of measurement in the past. A second principle is that these systems should provide capabilities for user-specifiable interpretation of process events, object state changes, and calendar time abstractions in order to allow a representative basis for project analysis. Another principle is that these systems should provide an extensible integration framework that supports the addition of new techniques and integrates approaches for synergistic application. These and other architectural principles are embodied in a prototype analysis and feedback system, called Amadeus. This prototype system is intended to demonstrate the feasibility and merit of these systems and the principles for designing them. An initial version of the system is running, and it has been used in several evaluative studies. We define the abstract interfaces in Amadeus and describe its operation. Metricdriven analysis and feedback systems influence the design of process modeling and programming languages and software environment architect ures. According] y,

Adversarial examples pose a serious threat to the robustness of machine learning models in genera... more Adversarial examples pose a serious threat to the robustness of machine learning models in general and deep learning models in particular. Computer vision tasks like image classification, facial recognition, object detection, etc. and natural language processing tasks like sentiment analysis and semantic similarity assessment have all been proven vulnerable to adversarial attacks. For computer vision tasks specifically, these carefully crafted perturbations to input images can cause targeted misclassifications to a label of the attacker’s choice, without the perturbations being detectable to the naked eye. A particular class of adversarial attacks called black box attacks can be used to fool a model under attack despite not having access to the model parameters or input datasets used to train the model. As part of the research presented in this paper, we first deploy a range of state of the art adversarial attacks against multiple face recognition pipelines trained in a black box se...

Testing commercial software is expensive and time consuming. Automated testing methods promise to... more Testing commercial software is expensive and time consuming. Automated testing methods promise to save a great deal of time and money throughout the software industry. One approach that is well-suited for the reactive systems found in telephone switching systems is specification-based testing. We have built a set of tools to automatically test software applications for violations of safety properties expressed in temporal logic. Our testing system automatically constructs finite state machine oracles corresponding to safety properties, builds test harnesses, and integrates them with an application. The test harness then generates inputs automatically to test the application.

We conducted a long-term experiment to compare the costs and benets of several dierent software i... more We conducted a long-term experiment to compare the costs and benets of several dierent software inspection methods. These methods were applied by professional developers to a commercial software product they were creating. Because the laboratory for this experiment was a live development eort, we took special care to minimize cost and risk to the project, while maximizing our ability to gather useful data. This article has several goals: (1) to describe the experiment's design and show how we used simula-tion techniques to optimize it, (2) to present our results and discuss their implications for both software practitioners and researchers, and (3) to discuss several new questions raised by our ndings. For each inspection we randomly assigned 3 independent variables: (1) the number of reviewers on each inspection team (1, 2 or 4), (2) the number of teams inspecting the code unit (1 or 2), and (3) the requirement that defects be repaired between the rst and second team's in...

many of its large-scale, mission-/safety-critical, and software-intensive national security syste... more many of its large-scale, mission-/safety-critical, and software-intensive national security systems using an open systems architecture (OSA) 1 approach, which leverages capable and reliable standards-based commercial off-the-shelf (COTS) software infrastructure components and modern software development practices for iterative and incremental development. As with other mission-/safety-critical domains, such as air traffic management, power grid, and automotive systems, a key goal of the Navy's OSA strategy has been to field and manage affordable and superior capabilities more rapidly at reduced cost. Although Navy OSA efforts have yielded some success, new challenges have arisen in the context of large-scale cyber-physical systems (CPSs), which play a critical role in existing and planned naval combat systems that combine computers, networks, and sensors to track and engage with enemy threats. This article describes the needs of — and advances in — software infrastructure manage...

ArXiv, 2021

Specified Certainty Classification (SCC) is a new paradigm for employing classifiers whose output... more Specified Certainty Classification (SCC) is a new paradigm for employing classifiers whose outputs carry uncertainties, typically in the form of Bayesian posterior probabilities. By allowing the classifier output to be less precise than one of a set of atomic decisions, SCC allows all decisions to achieve a specified level of certainty, as well as provides insights into classifier behavior by examining all decisions that are possible. Our primary illustration is read classification for reference-guided genome assembly, but we demonstrate the breadth of SCC by also analyzing COVID-19 vaccination data.

Proceedings of the eighteenth international symposium on Software testing and analysis, 2009

The increasing complexity of configurable software systems has created a need for more intelligen... more The increasing complexity of configurable software systems has created a need for more intelligent sampling mechanisms to detect and characterize failure-inducing dependencies between configurations. Prior work-in idealized environments-has shown that test schedules based on a mathematical object, called a covering array, in combination with classification techniques, can meet this need. Applying this approach in practice, however, is tricky because testing time and resource availability are unpredictable, and because failure characteristics can change from release to release. With current approaches developers must set a key covering array parameter (its strength) based on estimated release times and failure characterizations. This will influence the outcome of their results. In this paper we propose a new approach that incrementally builds covering array schedules. This approach begins at a low strength, and then iteratively increases strength as resources allow. At each stage previously tested configurations are reused, thus avoiding duplication of work. With the incremental approach developers need never commit to a specific covering array strength. Instead, by using progressively stronger covering array schedules, failures due to few configuration dependencies can be found and classified as soon and as cheaply as possibly. Additionally, it eliminates the risks of committing to overly strong test schedules. We evaluate this new approach through a case study on three consecutive releases of MySQL, an open source database. Our results suggest that our approach is as good or better than previous approaches, costing less in most cases, and allowing greater flexibility in environments with unpredictable development constraints.

Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, 2007

Proceedings of the 17th international conference on Software engineering - ICSE '95, 1995

Proceedings of the 19th international conference on Software engineering - ICSE '97, 1997

The dissemination of critical information and the synchronization of coordinated activities are c... more The dissemination of critical information and the synchronization of coordinated activities are critical problems in geographically separated, large-scale, software development. While these problems are not insurmountable, their solutions have v arying trade-o s in terms of time, cost and e ectiveness. Our previous studies have shown that the inspection interval is typically lengthened because of schedule con icts among inspectors which delay the usually required inspection collection meeting. We present and justify a solution using an intranet web that is both timely in its dissemination of information and e ective in its coordination of distributed inspectors. First, exploiting a naturally occurring experiment reported here, we conclude that the asynchronous collection of inspection results is at least as e ective a s the synchronous collection of those results. Second, exploiting the information dissemination qualities and the on-demand nature of information retrieval of the web, and the platform independence of browsers, we built an inexpensive tool that integrates seamlessly into the current development process. By seamless we mean an identical paper ow that results in an almost identical inspection process. The acceptance of the inspection tool has been excellent. The cost savings just from the reduction in paper work

The International Journal of High Performance Computing Applications, 2004

One key to improving high performance computing (HPC) productivity is to find better ways to meas... more One key to improving high performance computing (HPC) productivity is to find better ways to measure it. We define productivity in terms of mission goals, i.e. greater productivity means that more science is accomplished with less cost and effort. Traditional software productivity metrics and computing benchmarks have proven inadequate for assessing or predicting such end-to-end productivity. In this paper we introduce a new approach to measuring productivity in HPC applications that addresses both development time and execution time. Our goal is to develop a public repository of effective productivity benchmarks that anyone in the HPC community can apply to assess or predict productivity.

IEEE Transactions on Software Engineering, 2014

Software configurability has many benefits, but it also makes programs much harder to test, as in... more Software configurability has many benefits, but it also makes programs much harder to test, as in the worst case the program must be tested under every possible configuration. One potential remedy to this problem is combinatorial interaction testing (CIT), in which typically the developer selects a strength t and then computes a covering array containing all t-way configuration option combinations. However, in a prior study we showed that several programs have important highstrength interactions (combinations of a subset of configuration options) that CIT is highly unlikely to generate in practice. In this paper, we propose a new algorithm called interaction tree discovery (iTree) that aims to identify sets of configurations to test that are smaller than those generated by CIT, while also including important high-strength interactions missed by practical applications of CIT. On each iteration of iTree, we first use low-strength CIT to test the program under a set of configurations, and then apply machine learning techniques to discover new interactions that are potentially responsible for any new coverage seen. By repeating this process, iTree builds up a set of configurations likely to contain key high-strength interactions. We evaluated iTree by comparing the coverage it achieves versus covering arrays and randomly generated configuration sets. Our results strongly suggest that iTree can identify high-coverage sets of configurations more effectively than traditional CIT or random sampling.

IEEE Transactions on Software Engineering, 2007

IEEE Transactions on Software Engineering, 2009

Software engineers increasingly emphasize agility and flexibility in their designs and developmen... more Software engineers increasingly emphasize agility and flexibility in their designs and development approaches. They increasingly use distributed development teams, rely on component assembly and deployment rather than green field code writing, rapidly evolve the system through incremental development and frequent updating, and use flexible product designs supporting extensive end-user customization. While agility and flexibility have many benefits, they also create an enormous number of potential system configurations built from rapidly changing component implementations. Since today's quality assurance (QA) techniques do not scale to handle highly configurable systems, we are developing and validating novel software QA processes and tools that leverage the extensive computing resources of user and developer communities in a distributed, continuous manner to improve software quality significantly. This paper provides several contributions to the study of distributed, continuous QA (DCQA). First, it shows the structure and functionality of Skoll, which is an environment that defines a generic around-the-world, around-the-clock QA process and several sophisticated tools that support this process. Second, it describes several novel QA processes built using the Skoll environment. Third, it presents two studies using Skoll: one involving user testing of the Mozilla browser and another involving continuous build, integration, and testing of the ACE+TAO communication software package. The results of our studies suggest that the Skoll environment can manage and control distributed continuous QA processes more effectively than conventional QA processes. For example, our DCQA processes rapidly identified problems that had taken the ACE+TAO developers much longer to find and several of which they had not found. Moreover, the automatic analysis of QA results provided developers information that enabled them to quickly find the root cause of problems.

IEEE Transactions on Software Engineering, 2007

IEEE Transactions on Software Engineering, 2006

IEEE Transactions on Software Engineering, 2002

Configuration management is an important problem in large software systems. When dealing with hun... more Configuration management is an important problem in large software systems. When dealing with hundreds of components, keeping track of version changes and various dependency constraints imposed on the system, throughout its development life cycle is very challenging. Current approaches are ad hoc and proprietary, and there exists no standard for specifying valid software configurations. We propose a novel formalization for configuration management, based on the approaches developed in classic knowledge representation domain. Component constraints and version restrictions are encoded in an ontology using the standard OWL-DL language (a W3C recommendation), which facilitates the sharing of knowledge about configurations, across various systems. Detection and pinpointing of component inconsistencies, by human, is a painstaking and time consuming process. The machine readability of the OWL language enables us to apply reasoning on the specification, and automatically deduce the validity of test configurations. In addition, justifications on the validity of a configuration are provided.

Classification methods have troubles with missing data. Even CART, which was designed to deal wit... more Classification methods have troubles with missing data. Even CART, which was designed to deal with missing data, performs poorly when run with over 90% of the predictors unobserved. We use the Apriori algorithm to fit decision trees by converting the continuous predictors to categorical variables, bypassing the missing data problem by treating missing data as absent items. We demonstrate our methodology in a setting simulating a distributed, low-overhead, quality assurance system, where we have control over which predictors are missing for each observation. We also demonstrate how performance can be improved by the introduction of a simple adaptive sampling method.

[1991 Proceedings] 13th International Conference on Software Engineering

Metric-driven analysis and feedback systems enable developers to define empirically guided proces... more Metric-driven analysis and feedback systems enable developers to define empirically guided processes for software development and maintenance. These systems enable developers to achieve S. E.I. process maturit y levels 4 and 5, "managed" and "optimizing" processes. We define a set of architectural principles and abstract interfaces for designing metric-driven analysis and feedback systems. A fundamental principle is to make measurement active by integrating measurement and process, which contrasts with the primarily passive use of measurement in the past. A second principle is that these systems should provide capabilities for user-specifiable interpretation of process events, object state changes, and calendar time abstractions in order to allow a representative basis for project analysis. Another principle is that these systems should provide an extensible integration framework that supports the addition of new techniques and integrates approaches for synergistic application. These and other architectural principles are embodied in a prototype analysis and feedback system, called Amadeus. This prototype system is intended to demonstrate the feasibility and merit of these systems and the principles for designing them. An initial version of the system is running, and it has been used in several evaluative studies. We define the abstract interfaces in Amadeus and describe its operation. Metricdriven analysis and feedback systems influence the design of process modeling and programming languages and software environment architect ures. According] y,

Adversarial examples pose a serious threat to the robustness of machine learning models in genera... more Adversarial examples pose a serious threat to the robustness of machine learning models in general and deep learning models in particular. Computer vision tasks like image classification, facial recognition, object detection, etc. and natural language processing tasks like sentiment analysis and semantic similarity assessment have all been proven vulnerable to adversarial attacks. For computer vision tasks specifically, these carefully crafted perturbations to input images can cause targeted misclassifications to a label of the attacker’s choice, without the perturbations being detectable to the naked eye. A particular class of adversarial attacks called black box attacks can be used to fool a model under attack despite not having access to the model parameters or input datasets used to train the model. As part of the research presented in this paper, we first deploy a range of state of the art adversarial attacks against multiple face recognition pipelines trained in a black box se...

Testing commercial software is expensive and time consuming. Automated testing methods promise to... more Testing commercial software is expensive and time consuming. Automated testing methods promise to save a great deal of time and money throughout the software industry. One approach that is well-suited for the reactive systems found in telephone switching systems is specification-based testing. We have built a set of tools to automatically test software applications for violations of safety properties expressed in temporal logic. Our testing system automatically constructs finite state machine oracles corresponding to safety properties, builds test harnesses, and integrates them with an application. The test harness then generates inputs automatically to test the application.

We conducted a long-term experiment to compare the costs and benets of several dierent software i... more We conducted a long-term experiment to compare the costs and benets of several dierent software inspection methods. These methods were applied by professional developers to a commercial software product they were creating. Because the laboratory for this experiment was a live development eort, we took special care to minimize cost and risk to the project, while maximizing our ability to gather useful data. This article has several goals: (1) to describe the experiment's design and show how we used simula-tion techniques to optimize it, (2) to present our results and discuss their implications for both software practitioners and researchers, and (3) to discuss several new questions raised by our ndings. For each inspection we randomly assigned 3 independent variables: (1) the number of reviewers on each inspection team (1, 2 or 4), (2) the number of teams inspecting the code unit (1 or 2), and (3) the requirement that defects be repaired between the rst and second team's in...

many of its large-scale, mission-/safety-critical, and software-intensive national security syste... more many of its large-scale, mission-/safety-critical, and software-intensive national security systems using an open systems architecture (OSA) 1 approach, which leverages capable and reliable standards-based commercial off-the-shelf (COTS) software infrastructure components and modern software development practices for iterative and incremental development. As with other mission-/safety-critical domains, such as air traffic management, power grid, and automotive systems, a key goal of the Navy's OSA strategy has been to field and manage affordable and superior capabilities more rapidly at reduced cost. Although Navy OSA efforts have yielded some success, new challenges have arisen in the context of large-scale cyber-physical systems (CPSs), which play a critical role in existing and planned naval combat systems that combine computers, networks, and sensors to track and engage with enemy threats. This article describes the needs of — and advances in — software infrastructure manage...

ArXiv, 2021

Specified Certainty Classification (SCC) is a new paradigm for employing classifiers whose output... more Specified Certainty Classification (SCC) is a new paradigm for employing classifiers whose outputs carry uncertainties, typically in the form of Bayesian posterior probabilities. By allowing the classifier output to be less precise than one of a set of atomic decisions, SCC allows all decisions to achieve a specified level of certainty, as well as provides insights into classifier behavior by examining all decisions that are possible. Our primary illustration is read classification for reference-guided genome assembly, but we demonstrate the breadth of SCC by also analyzing COVID-19 vaccination data.

Proceedings of the eighteenth international symposium on Software testing and analysis, 2009

The increasing complexity of configurable software systems has created a need for more intelligen... more The increasing complexity of configurable software systems has created a need for more intelligent sampling mechanisms to detect and characterize failure-inducing dependencies between configurations. Prior work-in idealized environments-has shown that test schedules based on a mathematical object, called a covering array, in combination with classification techniques, can meet this need. Applying this approach in practice, however, is tricky because testing time and resource availability are unpredictable, and because failure characteristics can change from release to release. With current approaches developers must set a key covering array parameter (its strength) based on estimated release times and failure characterizations. This will influence the outcome of their results. In this paper we propose a new approach that incrementally builds covering array schedules. This approach begins at a low strength, and then iteratively increases strength as resources allow. At each stage previously tested configurations are reused, thus avoiding duplication of work. With the incremental approach developers need never commit to a specific covering array strength. Instead, by using progressively stronger covering array schedules, failures due to few configuration dependencies can be found and classified as soon and as cheaply as possibly. Additionally, it eliminates the risks of committing to overly strong test schedules. We evaluate this new approach through a case study on three consecutive releases of MySQL, an open source database. Our results suggest that our approach is as good or better than previous approaches, costing less in most cases, and allowing greater flexibility in environments with unpredictable development constraints.

Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, 2007

Proceedings of the 17th international conference on Software engineering - ICSE '95, 1995

Proceedings of the 19th international conference on Software engineering - ICSE '97, 1997

The dissemination of critical information and the synchronization of coordinated activities are c... more The dissemination of critical information and the synchronization of coordinated activities are critical problems in geographically separated, large-scale, software development. While these problems are not insurmountable, their solutions have v arying trade-o s in terms of time, cost and e ectiveness. Our previous studies have shown that the inspection interval is typically lengthened because of schedule con icts among inspectors which delay the usually required inspection collection meeting. We present and justify a solution using an intranet web that is both timely in its dissemination of information and e ective in its coordination of distributed inspectors. First, exploiting a naturally occurring experiment reported here, we conclude that the asynchronous collection of inspection results is at least as e ective a s the synchronous collection of those results. Second, exploiting the information dissemination qualities and the on-demand nature of information retrieval of the web, and the platform independence of browsers, we built an inexpensive tool that integrates seamlessly into the current development process. By seamless we mean an identical paper ow that results in an almost identical inspection process. The acceptance of the inspection tool has been excellent. The cost savings just from the reduction in paper work

The International Journal of High Performance Computing Applications, 2004

One key to improving high performance computing (HPC) productivity is to find better ways to meas... more One key to improving high performance computing (HPC) productivity is to find better ways to measure it. We define productivity in terms of mission goals, i.e. greater productivity means that more science is accomplished with less cost and effort. Traditional software productivity metrics and computing benchmarks have proven inadequate for assessing or predicting such end-to-end productivity. In this paper we introduce a new approach to measuring productivity in HPC applications that addresses both development time and execution time. Our goal is to develop a public repository of effective productivity benchmarks that anyone in the HPC community can apply to assess or predict productivity.

IEEE Transactions on Software Engineering, 2014

Software configurability has many benefits, but it also makes programs much harder to test, as in... more Software configurability has many benefits, but it also makes programs much harder to test, as in the worst case the program must be tested under every possible configuration. One potential remedy to this problem is combinatorial interaction testing (CIT), in which typically the developer selects a strength t and then computes a covering array containing all t-way configuration option combinations. However, in a prior study we showed that several programs have important highstrength interactions (combinations of a subset of configuration options) that CIT is highly unlikely to generate in practice. In this paper, we propose a new algorithm called interaction tree discovery (iTree) that aims to identify sets of configurations to test that are smaller than those generated by CIT, while also including important high-strength interactions missed by practical applications of CIT. On each iteration of iTree, we first use low-strength CIT to test the program under a set of configurations, and then apply machine learning techniques to discover new interactions that are potentially responsible for any new coverage seen. By repeating this process, iTree builds up a set of configurations likely to contain key high-strength interactions. We evaluated iTree by comparing the coverage it achieves versus covering arrays and randomly generated configuration sets. Our results strongly suggest that iTree can identify high-coverage sets of configurations more effectively than traditional CIT or random sampling.

IEEE Transactions on Software Engineering, 2007

IEEE Transactions on Software Engineering, 2009

Software engineers increasingly emphasize agility and flexibility in their designs and developmen... more Software engineers increasingly emphasize agility and flexibility in their designs and development approaches. They increasingly use distributed development teams, rely on component assembly and deployment rather than green field code writing, rapidly evolve the system through incremental development and frequent updating, and use flexible product designs supporting extensive end-user customization. While agility and flexibility have many benefits, they also create an enormous number of potential system configurations built from rapidly changing component implementations. Since today's quality assurance (QA) techniques do not scale to handle highly configurable systems, we are developing and validating novel software QA processes and tools that leverage the extensive computing resources of user and developer communities in a distributed, continuous manner to improve software quality significantly. This paper provides several contributions to the study of distributed, continuous QA (DCQA). First, it shows the structure and functionality of Skoll, which is an environment that defines a generic around-the-world, around-the-clock QA process and several sophisticated tools that support this process. Second, it describes several novel QA processes built using the Skoll environment. Third, it presents two studies using Skoll: one involving user testing of the Mozilla browser and another involving continuous build, integration, and testing of the ACE+TAO communication software package. The results of our studies suggest that the Skoll environment can manage and control distributed continuous QA processes more effectively than conventional QA processes. For example, our DCQA processes rapidly identified problems that had taken the ACE+TAO developers much longer to find and several of which they had not found. Moreover, the automatic analysis of QA results provided developers information that enabled them to quickly find the root cause of problems.

IEEE Transactions on Software Engineering, 2007

IEEE Transactions on Software Engineering, 2006

IEEE Transactions on Software Engineering, 2002

Configuration management is an important problem in large software systems. When dealing with hun... more Configuration management is an important problem in large software systems. When dealing with hundreds of components, keeping track of version changes and various dependency constraints imposed on the system, throughout its development life cycle is very challenging. Current approaches are ad hoc and proprietary, and there exists no standard for specifying valid software configurations. We propose a novel formalization for configuration management, based on the approaches developed in classic knowledge representation domain. Component constraints and version restrictions are encoded in an ontology using the standard OWL-DL language (a W3C recommendation), which facilitates the sharing of knowledge about configurations, across various systems. Detection and pinpointing of component inconsistencies, by human, is a painstaking and time consuming process. The machine readability of the OWL language enables us to apply reasoning on the specification, and automatically deduce the validity of test configurations. In addition, justifications on the validity of a configuration are provided.

Classification methods have troubles with missing data. Even CART, which was designed to deal wit... more Classification methods have troubles with missing data. Even CART, which was designed to deal with missing data, performs poorly when run with over 90% of the predictors unobserved. We use the Apriori algorithm to fit decision trees by converting the continuous predictors to categorical variables, bypassing the missing data problem by treating missing data as absent items. We demonstrate our methodology in a setting simulating a distributed, low-overhead, quality assurance system, where we have control over which predictors are missing for each observation. We also demonstrate how performance can be improved by the introduction of a simple adaptive sampling method.