Kenneth Kotovsky - Academia.edu (original) (raw)

Papers by Kenneth Kotovsky

In this paper we examine the role played by working memory demands in determining problem difficu... more In this paper we examine the role played by working memory demands in determining problem difficulty during the solution of Tower of Hanoi Problem isomorphs. We do so by describing a production system model that accounts for subjects' performance on these problems via a dynamic analysis of the memory load imposed by the problem and of changes in that load during the problem solving episode. We also present the results of detailed testing of the model against human subject data. The model uses a highly constrained working memory to account for a number of features of the problem solving behavior, including the dichotomous (exploratory and final path) nature of the problem solving, the relative difficulty of the problems, the particular moves made in each state of the problem space, and the temporal patterning of the final path moves.

Designers must often create solutions to problems that exhibit dynamic characteristics. For insta... more Designers must often create solutions to problems that exhibit dynamic characteristics. For instance, a client might modify specifications after design has commenced, or a competitor may introduce a new technology or feature. This paper presents a cognitive study that was conducted to explore the manner in which design teams respond to such situations. In the study, teams of undergraduate engineering students sought to solve a design task that was subject to two large, unexpected changes in problem formulation that were introduced during solving. High-and lowperforming teams demonstrated very different approaches to solving the problem and overcoming the changes. The results indicate that there may exist a relationship between problem characteristics and fruitful solution strategies.

Insights uncovered by research in design cognition are often utilized to develop methods used by ... more Insights uncovered by research in design cognition are often utilized to develop methods used by human designers; in this work such insights are used to inform and improve computational methodologies. This paper introduces the Heterogeneous Simulated Annealing Team (HSAT) algorithm, a multi-agent simulated annealing algorithm. HSAT is based on a validated computational model of human-based MD-15-1409 Cagan 2 engineering design, and retains characteristics of the model that structure interaction between team members and allow for heterogeneous search strategies to be employed within a team. The performance of this new algorithm is compared to several other simulated annealing based algorithms on three carefully selected benchmarking functions. The HSAT algorithm provides terminal solutions that are better on average than other algorithms explored in this work.

Many design tasks are subject to changes in goals or constraints. For instance, a client might mo... more Many design tasks are subject to changes in goals or constraints. For instance, a client might modify specifications after design has commenced, or a competitor may introduce a new technology or feature. A design team often cannot anticipate such changes, yet they pose a considerable challenge. This paper presents a study where engineering teams sought to solve a design task that was subject to two large, unexpected changes in problem formulation that occurred during problem solving. Continuous design data was collected to observe how the designers responded to the changes. We show that high-and low-performing teams demonstrated very different approaches to solving the problem and overcoming the changes. In particular, high-performing teams achieved simple designs and extensively explored small portions of the design space; lowperforming teams explored complex designs with little exploration around a target area of the design space. These strategic differences are interpreted with respect to cognitive load theory and goal theory. The results raise questions as to the relationship between characteristics of design problems and solution strategies. In addition, an attempt at increasing the teams' resilience in the face of unexpected changes is introduced by encouraging early divergent search.

Computers in Human Behavior, Feb 1, 2022

Data in Brief, Apr 1, 2022

Human subject experiments are performed to evaluate the influence of artificial intelligence (AI)... more Human subject experiments are performed to evaluate the influence of artificial intelligence (AI) process management on human design teams solving a complex engineering problem and compare that to the influence of human process management. Participants are grouped into teams of five individuals and asked to generate a drone fleet and plan routes to deliver parcels to a given customer market. The teams are placed under the guidance of either a human or an AI external process manager. Halfway through the experiment, the customer market is changed unexpectedly, requiring teams to adjust their strategy. During the experiment, participants can create, evaluate, share their drone designs and delivery routes, and communicate with their team through a text chat tool using a collaborative research platform called HyForm. The research platform collects step-by-step logs of the actions made by and communication amongst participants in both the design team's roles and the process managers. This article presents the data sets collected for 171 participants assigned to 31 design teams, 15 teams under the guidance of an AI agent (5 participants), and 16 teams under the guidance of a human manager (6 participants). These data sets can be used for data-driven design, behavioral analyses, sequence-based analyses, and natural language processing.

Computers in Human Behavior, Feb 1, 2023

A commonly held presumption is that the production of a team is superior to that of individual pe... more A commonly held presumption is that the production of a team is superior to that of individual performance. However, in certain scenarios, such as during brainstorming activities and in configuration engineering design problems, it has been shown that individuals working alone are more effective than teams working together. This research considers whether the same outcomes hold for a more open-ended scenario, in conceptual engineering design. Thus, a behavioral study is run with freshman engineering students solving a conceptual design problem working in teams or individually. Results corroborate previous findings, showing that individuals outperform teams in the quality of their design solutions. One of the primary differences between individuals and group problem solving is the fact that groups need to verbalize to communicate ideas. Consequently, this study also analyzes how verbalization, which may be one disadvantage of team problem solving, affects the performance of individuals in this context of conceptual engineering design. Individuals who verbalize throughout problem solving, however, perform similarly to those who did not. Overall, the results from this study suggest that, individuals are still better performers and teams may not always be the optimal circumstance. Moreover, verbalization does not seem to act as a cognitive barrier to problem solving, and further investigation needs to be done to diagnose the potential impediments which put teams at a disadvantage to individuals during conceptual design.

Novel design methodologies are often evaluated through empirical studies involving human designer... more Novel design methodologies are often evaluated through empirical studies involving human designers. However, such empirical studies can incur a high personnel cost. Further, it can be difficult to isolate the effects of specific team or individual characteristics. These limitations could be bypassed by employing a computational model of design teams. This work introduces the Cognitively-Inspired Simulated Annealing Teams (CISAT) modeling framework, an agent-based platform that provides a means for efficiently simulating human design teams. A number of empirically demonstrated cognitive phenomena are modeled within the platform, striking a balance between model simplicity and direct applicability to engineering design problems. This paper discusses the composition of the CISAT modeling framework and demonstrates how it can be used to simulate the performance of human design teams in a cognitive study. Results simulated with CISAT are compared directly to the results derived from human designers. Finally, the CISAT model is also used to investigate the characteristics that were most and least helpful to teams during the cognitive study.

Thinking & Reasoning, Aug 9, 2021

Creative problem solving is often conceptualised as a process of search. However, little is known... more Creative problem solving is often conceptualised as a process of search. However, little is known about the difficulties of carrying out this search process. We conducted three studies examining ho...

A team's design-the structuring of its resources and flows of knowledge-is an important element d... more A team's design-the structuring of its resources and flows of knowledge-is an important element determining its effectiveness. An essential element in achieving a team's problemsolving potential is the role that interdependence, in both the task and the organization, plays in determining the dynamic and emergent system-level properties of the organization. In this paper, we present a computational platform for experimentally investigating the influence of informational dependencies found in the design of a complex system for exploring their role in determining system behaviors and performance. The approach presented in this paper is a multiagent simulation of the conceptual design of space mission plans by Team X, an advanced project design group at NASA's Jet Propulsion Laboratory. The algorithm is composed of rich descriptive models of both the team-types and timing of interactions, collaborative methods, sequencing, rates of convergence-and the task-primary variables, their behaviors and relations, and the approaches used to resolve them. The objective is to create an environment of interaction representative of that found in actual design sessions. Better understanding how the dynamics arising from organizational and domain interdependencies impact an organization's ability to effectively resolve its task should lead to the development of guidelines for better coping with task complexities, suggest ways to better design organizations, as well as suggest ways for improving the search for innovative solutions.

Data in Brief, Feb 1, 2023

Proceedings of the Design Society, May 1, 2022

For successful human-artificial intelligence (AI) collaboration in design, human designers must p... more For successful human-artificial intelligence (AI) collaboration in design, human designers must properly use AI input. Some factors affecting that use are designers' self-confidence and competence and those variables' impact on reliance on AI. This work studies how designers' self-confidence before and during teamwork and overall competence are associated with their performance as teammates, measured by AI reliance and overall team score. Results show that designers' self-confidence and competence have very different impacts on their collaborative performance depending on the accuracy of AI.

Journal of Mechanical Design

This work introduces the Proficient Simulated Annealing Design Agent Model (PSADA), a cognitively... more This work introduces the Proficient Simulated Annealing Design Agent Model (PSADA), a cognitively inspired, agent-based model of engineering configuration design. PSADA models different proficiency agents using move selection heuristics and problem space search strategies, both of which are identified and extracted from prior human subject studies. The model is validated with two design problems. Agents are compared to human designers and show the accurate simulation of the behaviors of the different proficiency designers. These behavior differences lead to significantly different performance levels, matching the human performance levels with just one exception. These validated heterogeneous agents are placed into teams and confirmed previous findings that the most proficient member of a configuration design team has the largest impact (positive or negative) on team performance. The PSADA model is introduced as a scalable platform to further explore configuration design proficiency’...

Decision Support Systems, May 1, 2021

Proceedings of the Design Society, May 1, 2022

This work studies the perception of the impacts of AI and human process managers during a complex... more This work studies the perception of the impacts of AI and human process managers during a complex design task. Although performance and perceptions by teams that are AI-versus human-managed are similar, we show that how team members discern the identity of their process manager (human/AI), impacts their perceptions. They discern the interventions as significantly more helpful and manager sensitive to the needs of the team, if they believe to be managed by a human. Further results provide deeper insights into automating real-time process management and the efficacy of AI to fill that role.

Journal of Mechanical Design, Jun 30, 2022

This brief extends prior research by the authors on studying the impacts of interventions provide... more This brief extends prior research by the authors on studying the impacts of interventions provided by either a human or an artificial intelligence (AI) process manager on team behaviors. Our earlier research found that a created AI process manager matched the capabilities of human process management. Here, these data are studied further to identify the impact of different types of interventions on team behaviors and outcomes. This deeper dive is done via two unique perspectives: comparing teams’ problem-solving processes before and after interventions are provided, and through a regression analysis between intervention counts and performance. Results show overall mixed adherence to the provided interventions, and that this adherence also depends on the intervention type. The most significant impact on the team process arises from the communication frequency interventions. Furthermore, a regression analysis identifies the interventions with the greatest correlation with team performance, indicating a better selection of interventions from the AI process manager. Paired together, the results show the feasibility of automated process management via AI and shed light on the effective implementation of intervention strategies for future development and deployment.

In this paper we examine the role played by working memory demands in determining problem difficu... more In this paper we examine the role played by working memory demands in determining problem difficulty during the solution of Tower of Hanoi Problem isomorphs. We do so by describing a production system model that accounts for subjects' performance on these problems via a dynamic analysis of the memory load imposed by the problem and of changes in that load during the problem solving episode. We also present the results of detailed testing of the model against human subject data. The model uses a highly constrained working memory to account for a number of features of the problem solving behavior, including the dichotomous (exploratory and final path) nature of the problem solving, the relative difficulty of the problems, the particular moves made in each state of the problem space, and the temporal patterning of the final path moves.

Designers must often create solutions to problems that exhibit dynamic characteristics. For insta... more Designers must often create solutions to problems that exhibit dynamic characteristics. For instance, a client might modify specifications after design has commenced, or a competitor may introduce a new technology or feature. This paper presents a cognitive study that was conducted to explore the manner in which design teams respond to such situations. In the study, teams of undergraduate engineering students sought to solve a design task that was subject to two large, unexpected changes in problem formulation that were introduced during solving. High-and lowperforming teams demonstrated very different approaches to solving the problem and overcoming the changes. The results indicate that there may exist a relationship between problem characteristics and fruitful solution strategies.

Insights uncovered by research in design cognition are often utilized to develop methods used by ... more Insights uncovered by research in design cognition are often utilized to develop methods used by human designers; in this work such insights are used to inform and improve computational methodologies. This paper introduces the Heterogeneous Simulated Annealing Team (HSAT) algorithm, a multi-agent simulated annealing algorithm. HSAT is based on a validated computational model of human-based MD-15-1409 Cagan 2 engineering design, and retains characteristics of the model that structure interaction between team members and allow for heterogeneous search strategies to be employed within a team. The performance of this new algorithm is compared to several other simulated annealing based algorithms on three carefully selected benchmarking functions. The HSAT algorithm provides terminal solutions that are better on average than other algorithms explored in this work.

Many design tasks are subject to changes in goals or constraints. For instance, a client might mo... more Many design tasks are subject to changes in goals or constraints. For instance, a client might modify specifications after design has commenced, or a competitor may introduce a new technology or feature. A design team often cannot anticipate such changes, yet they pose a considerable challenge. This paper presents a study where engineering teams sought to solve a design task that was subject to two large, unexpected changes in problem formulation that occurred during problem solving. Continuous design data was collected to observe how the designers responded to the changes. We show that high-and low-performing teams demonstrated very different approaches to solving the problem and overcoming the changes. In particular, high-performing teams achieved simple designs and extensively explored small portions of the design space; lowperforming teams explored complex designs with little exploration around a target area of the design space. These strategic differences are interpreted with respect to cognitive load theory and goal theory. The results raise questions as to the relationship between characteristics of design problems and solution strategies. In addition, an attempt at increasing the teams' resilience in the face of unexpected changes is introduced by encouraging early divergent search.

Computers in Human Behavior, Feb 1, 2022

Data in Brief, Apr 1, 2022

Human subject experiments are performed to evaluate the influence of artificial intelligence (AI)... more Human subject experiments are performed to evaluate the influence of artificial intelligence (AI) process management on human design teams solving a complex engineering problem and compare that to the influence of human process management. Participants are grouped into teams of five individuals and asked to generate a drone fleet and plan routes to deliver parcels to a given customer market. The teams are placed under the guidance of either a human or an AI external process manager. Halfway through the experiment, the customer market is changed unexpectedly, requiring teams to adjust their strategy. During the experiment, participants can create, evaluate, share their drone designs and delivery routes, and communicate with their team through a text chat tool using a collaborative research platform called HyForm. The research platform collects step-by-step logs of the actions made by and communication amongst participants in both the design team's roles and the process managers. This article presents the data sets collected for 171 participants assigned to 31 design teams, 15 teams under the guidance of an AI agent (5 participants), and 16 teams under the guidance of a human manager (6 participants). These data sets can be used for data-driven design, behavioral analyses, sequence-based analyses, and natural language processing.

Computers in Human Behavior, Feb 1, 2023

A commonly held presumption is that the production of a team is superior to that of individual pe... more A commonly held presumption is that the production of a team is superior to that of individual performance. However, in certain scenarios, such as during brainstorming activities and in configuration engineering design problems, it has been shown that individuals working alone are more effective than teams working together. This research considers whether the same outcomes hold for a more open-ended scenario, in conceptual engineering design. Thus, a behavioral study is run with freshman engineering students solving a conceptual design problem working in teams or individually. Results corroborate previous findings, showing that individuals outperform teams in the quality of their design solutions. One of the primary differences between individuals and group problem solving is the fact that groups need to verbalize to communicate ideas. Consequently, this study also analyzes how verbalization, which may be one disadvantage of team problem solving, affects the performance of individuals in this context of conceptual engineering design. Individuals who verbalize throughout problem solving, however, perform similarly to those who did not. Overall, the results from this study suggest that, individuals are still better performers and teams may not always be the optimal circumstance. Moreover, verbalization does not seem to act as a cognitive barrier to problem solving, and further investigation needs to be done to diagnose the potential impediments which put teams at a disadvantage to individuals during conceptual design.

Novel design methodologies are often evaluated through empirical studies involving human designer... more Novel design methodologies are often evaluated through empirical studies involving human designers. However, such empirical studies can incur a high personnel cost. Further, it can be difficult to isolate the effects of specific team or individual characteristics. These limitations could be bypassed by employing a computational model of design teams. This work introduces the Cognitively-Inspired Simulated Annealing Teams (CISAT) modeling framework, an agent-based platform that provides a means for efficiently simulating human design teams. A number of empirically demonstrated cognitive phenomena are modeled within the platform, striking a balance between model simplicity and direct applicability to engineering design problems. This paper discusses the composition of the CISAT modeling framework and demonstrates how it can be used to simulate the performance of human design teams in a cognitive study. Results simulated with CISAT are compared directly to the results derived from human designers. Finally, the CISAT model is also used to investigate the characteristics that were most and least helpful to teams during the cognitive study.

Thinking & Reasoning, Aug 9, 2021

Creative problem solving is often conceptualised as a process of search. However, little is known... more Creative problem solving is often conceptualised as a process of search. However, little is known about the difficulties of carrying out this search process. We conducted three studies examining ho...

A team's design-the structuring of its resources and flows of knowledge-is an important element d... more A team's design-the structuring of its resources and flows of knowledge-is an important element determining its effectiveness. An essential element in achieving a team's problemsolving potential is the role that interdependence, in both the task and the organization, plays in determining the dynamic and emergent system-level properties of the organization. In this paper, we present a computational platform for experimentally investigating the influence of informational dependencies found in the design of a complex system for exploring their role in determining system behaviors and performance. The approach presented in this paper is a multiagent simulation of the conceptual design of space mission plans by Team X, an advanced project design group at NASA's Jet Propulsion Laboratory. The algorithm is composed of rich descriptive models of both the team-types and timing of interactions, collaborative methods, sequencing, rates of convergence-and the task-primary variables, their behaviors and relations, and the approaches used to resolve them. The objective is to create an environment of interaction representative of that found in actual design sessions. Better understanding how the dynamics arising from organizational and domain interdependencies impact an organization's ability to effectively resolve its task should lead to the development of guidelines for better coping with task complexities, suggest ways to better design organizations, as well as suggest ways for improving the search for innovative solutions.

Data in Brief, Feb 1, 2023

Proceedings of the Design Society, May 1, 2022

For successful human-artificial intelligence (AI) collaboration in design, human designers must p... more For successful human-artificial intelligence (AI) collaboration in design, human designers must properly use AI input. Some factors affecting that use are designers' self-confidence and competence and those variables' impact on reliance on AI. This work studies how designers' self-confidence before and during teamwork and overall competence are associated with their performance as teammates, measured by AI reliance and overall team score. Results show that designers' self-confidence and competence have very different impacts on their collaborative performance depending on the accuracy of AI.

Journal of Mechanical Design

This work introduces the Proficient Simulated Annealing Design Agent Model (PSADA), a cognitively... more This work introduces the Proficient Simulated Annealing Design Agent Model (PSADA), a cognitively inspired, agent-based model of engineering configuration design. PSADA models different proficiency agents using move selection heuristics and problem space search strategies, both of which are identified and extracted from prior human subject studies. The model is validated with two design problems. Agents are compared to human designers and show the accurate simulation of the behaviors of the different proficiency designers. These behavior differences lead to significantly different performance levels, matching the human performance levels with just one exception. These validated heterogeneous agents are placed into teams and confirmed previous findings that the most proficient member of a configuration design team has the largest impact (positive or negative) on team performance. The PSADA model is introduced as a scalable platform to further explore configuration design proficiency’...

Decision Support Systems, May 1, 2021

Proceedings of the Design Society, May 1, 2022

This work studies the perception of the impacts of AI and human process managers during a complex... more This work studies the perception of the impacts of AI and human process managers during a complex design task. Although performance and perceptions by teams that are AI-versus human-managed are similar, we show that how team members discern the identity of their process manager (human/AI), impacts their perceptions. They discern the interventions as significantly more helpful and manager sensitive to the needs of the team, if they believe to be managed by a human. Further results provide deeper insights into automating real-time process management and the efficacy of AI to fill that role.

Journal of Mechanical Design, Jun 30, 2022

This brief extends prior research by the authors on studying the impacts of interventions provide... more This brief extends prior research by the authors on studying the impacts of interventions provided by either a human or an artificial intelligence (AI) process manager on team behaviors. Our earlier research found that a created AI process manager matched the capabilities of human process management. Here, these data are studied further to identify the impact of different types of interventions on team behaviors and outcomes. This deeper dive is done via two unique perspectives: comparing teams’ problem-solving processes before and after interventions are provided, and through a regression analysis between intervention counts and performance. Results show overall mixed adherence to the provided interventions, and that this adherence also depends on the intervention type. The most significant impact on the team process arises from the communication frequency interventions. Furthermore, a regression analysis identifies the interventions with the greatest correlation with team performance, indicating a better selection of interventions from the AI process manager. Paired together, the results show the feasibility of automated process management via AI and shed light on the effective implementation of intervention strategies for future development and deployment.