Paulo Shakarian | Arizona State University (original) (raw)

Books by Paulo Shakarian

AAAI-MAKE, 2023

We study the performance of a commercially available large language model (LLM) known as ChatGPT ... more We study the performance of a commercially available large language model (LLM) known as ChatGPT on math word problems (MWPs) from the dataset DRAW-1K. To our knowledge, this is the first independent evaluation of ChatGPT. We found that ChatGPT's performance changes dramatically based on the requirement to show its work, failing 20% of the time when it provides work compared with 84% when it does not. Further several factors about MWPs relating to the number of unknowns and number of operations that lead to a higher probability of failure when compared with the prior, specifically noting (across all experiments) that the probability of failure increases linearly with the number of addition and subtraction operations. We also have released the dataset of ChatGPT's responses to the MWPs to support further work on the characterization of LLM performance and present baseline machine learning models to predict if ChatGPT can correctly answer an MWP. We have released a dataset comprised of ChatGPT's responses to support further research in this area.

ICLP, 2024

The ability to generate artificial human movement patterns while meeting location and time constr... more The ability to generate artificial human movement patterns while meeting location and time constraints is an important problem in the security community, particularly as it enables the study of the analog problem of detecting such patterns while maintaining privacy. We frame this problem as an instance of abduction guided by a novel parsimony function represented as an aggregate truth value over an annotated logic program. This approach has the added benefit of affording explainability to an analyst user. By showing that any subset of such a program can provide a lower bound on this parsimony requirement, we are able to abduce movement trajectories efficiently through an informed (i.e., A*) search. We describe how our implementation was enhanced with the application of multiple techniques in order to be scaled and integrated with a cloud-based software stack that included bottom-up rule learning, geolocated knowledge graph retrieval/management, and interfaces with government systems for independently conducted government-run tests for which we provide results. We also report on our own experiments showing that we not only provide exact results but also scale to very large scenarios and provide realistic agent trajectories that can go undetected by machine learning anomaly detectors.

In recent years, research on diffusion process in social networks has grown in a variety of field... more In recent years, research on diffusion process in social networks has grown in a variety of fields including computer science, physics, and biology. However, often times research in these individual disciplines becomes stove-piped. In this book, we focus on cutting-edge research in social network diffusion bringing together a range of ideas from these disciplines with the goal of creating a single volume that examines these ideas.

We sought to cover many of the most important concepts, models, and methods from these areas. We felt by exploring a variety of work from different fields that we could help open the door to more innovative findings in this fascinating area of diffusion in social networks.

Preprint currently available - book to be released by Springer in Fall, 2015.

Sushil Jajodia, Paulo Shakarian, VS Subrahmanian, Vipin Swarup, Cliff Wang This book features ... more Sushil Jajodia, Paulo Shakarian, VS Subrahmanian, Vipin Swarup, Cliff Wang

This book features a wide spectrum of the latest computer science research relating to cyber warfare, including military and policy dimensions. It is the first book to explore the scientific foundation of cyber warfare and features research from the areas of artificial intelligence, game theory, programming languages, graph theory and more. The high-level approach and emphasis on scientific rigor provides insights on ways to improve cyber warfare defense worldwide. Cyber Warfare: Building the Scientific Foundation targets researchers and practitioners working in cyber security, especially government employees or contractors. Advanced-level students in computer science and electrical engineering with an interest in security will also find this content valuable as a secondary textbook or reference.

• Provides a multidisciplinary approach to Cyber Warfare analyzing the information technology, mi... more • Provides a multidisciplinary approach to Cyber Warfare analyzing the information technology, military, policy, social, and scientific issues that are in play.
• Presents detailed case studies of cyber-attack including inter-state cyber-conflict (Russia-Estonia), cyber-attack as an element of an information operations strategy (Israel-Hezbollah,) and cyber-attack as a tool against dissidents within a state (Russia, Iran)
• Explores cyber-attack conducted by large, powerful, non-state hacking organizations such as Anonymous and LulzSec
• Covers cyber-attacks directed against infrastructure such including but not limited to water treatment plants, power-grid and a detailed account on Stuxent"""

Imagine yourself as a military officer in a conflict zone trying to identify locations of weapons... more Imagine yourself as a military officer in a conflict zone trying to identify locations of weapons caches supporting road-side bomb attacks on your country’s troops. Or imagine yourself as a public health expert trying to identify the location of contaminated water that is causing diarrheal diseases in a local population. Geospatial abduction is a new technique introduced by the authors that allows such problems to be solved. Geospatial Abduction provides the mathematics underlying geospatial abduction and the algorithms to solve them in practice; it has wide applicability and can be used by practitioners and researchers in many different fields. Real-world applications of geospatial abduction to military problems are included. Compelling examples drawn from other domains as diverse as criminology, epidemiology and archaeology are covered as well. This book also includes access to a dedicated website on geospatial abduction hosted by University of Maryland. Geospatial Abduction targets practitioners working in general AI, game theory, linear programming, data mining, machine learning, and more. Those working in the fields of computer science, mathematics, geoinformation, geological and biological science will also find this book valuable.

Papers by Paulo Shakarian

Big Data and Cognitive Computing

This article is an open access article distributed under the terms and conditions of the Creative... more This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY

The modeling of cascade processes in multi-agent systems in the form of complex networks has in r... more The modeling of cascade processes in multi-agent systems in the form of complex networks has in recent years become an important topic of study due to its many applications: the adoption of commercial products, spread of disease, the diffusion of an idea, etc. In this paper, we begin by identifying a desiderata of seven properties that a framework for modeling such processes should satisfy: the ability to represent attributes of both nodes and edges, an explicit representation of time, the ability to represent non-Markovian temporal relationships, representation of uncertain information, the ability to represent competing cascades, allowance of non-monotonic diffusion, and computational tractability. We then present the MANCaLog language, a formalism based on logic programming that satisfies all these desiderata, and focus on algorithms for finding minimal models (from which the outcome of cascades can be obtained) as well as how this formalism can be applied in real world scenarios. We are not aware of any other formalism in the literature that meets all of the above requirements.

Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '15, 2015

Elham Shaabani, Ashkan Aleali, Paulo Shakarian, John Bertetto KDD 2015 (Aug., 2015) Gang violence... more Elham Shaabani, Ashkan Aleali, Paulo Shakarian, John Bertetto KDD 2015 (Aug., 2015) Gang violence is a major problem in the United States ac- counting for a large fraction of homicides and other vio- lent crime. In this paper, we study the problem of early identification of violent gang members. Our approach re- lies on modified centrality measures that take into account additional data of the individuals in the social network of co-arrestees which together with other arrest metadata pro- vide a rich set of features for a classification algorithm. We show our approach obtains high precision and recall (0.89 and 0.78 respectively) in the case where the entire network is known and out-performs current approaches used by law- enforcement to the problem in the case where the network is discovered overtime by virtue of new arrests - mimick- ing real-world law-enforcement operations. Operational is- sues are also discussed as we are preparing to leverage this method in an operational environment.

SpringerBriefs in Computer Science, 2015

Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2015 - ASONAM '15, 2015

Ruocheng Guo, Elham Shaabani, Abhinav Bhatnagar, Paulo Shakarian ASONAM '15 When a piece ... more Ruocheng Guo, Elham Shaabani, Abhinav Bhatnagar, Paulo Shakarian ASONAM '15 When a piece of information (microblog, photograph, video, link, etc.) starts to spread in a social network, an important question arises: will it spread to "viral" proportions -- where "viral" is defined as an order-of-magnitude increase. However, several previous studies have established that cascade size and frequency are related through a power-law - which leads to a severe imbalance in this classification problem. In this paper, we devise a suite of measurements based on "structural diversity" -- the variety of social contexts (communities) in which individuals partaking in a given cascade engage. We demonstrate these measures are able to distinguish viral from non-viral cascades, despite the severe imbalance of the data for this problem. Further, we leverage these measurements as features in a classification approach, successfully predicting microblogs that grow from 50 to 500 reposts with precision of 0.69 and recall of 0.52 for the viral class - despite this class comprising under 2\% of samples. This significantly outperforms our baseline approach as well as the current state-of-the-art. Our work also demonstrates how we can tradeoff between precision and recall.

SpringerBriefs in Computer Science, 2015

(Originally Published in NGA Pathfinder, 2016) Analyzing large sets of data using techniques such... more (Originally Published in NGA Pathfinder, 2016)
Analyzing large sets of data using techniques such as machine learning, artificial intelligence and statistics can lead to improved insight and potentially better decisions. But the field is still young, and there are many challenges that must be considered along the way. Of specific concern are cleaning and preparation, causality, model transparency and decision support.

AAAI-MAKE, 2023

ICLP, 2024

Preprint currently available - book to be released by Springer in Fall, 2015.

Sushil Jajodia, Paulo Shakarian, VS Subrahmanian, Vipin Swarup, Cliff Wang This book features ... more Sushil Jajodia, Paulo Shakarian, VS Subrahmanian, Vipin Swarup, Cliff Wang

Big Data and Cognitive Computing

Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '15, 2015

SpringerBriefs in Computer Science, 2015

Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2015 - ASONAM '15, 2015

SpringerBriefs in Computer Science, 2015

Military Review, Jul 1, 2011

SpringerBriefs in Computer Science, 2015

Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '15, 2015

Introduction to Cyber-warfare, 2013

Mining Key-Hackers on Darkweb Forums, 2018

Recently, there is an interest in studying cyber crime from a hacker-centric perspective, whose i... more Recently, there is an interest in studying cyber crime from a hacker-centric perspective, whose insight is to locate key-hackers and use them to find credible threat intelligence. However, the great majority of users present in hacking environments seem to be unskilled or have fleeting interests, making the identification of key-hackers a complex problem. Moreover, as ground truth information is rare in this context, there is a lack of a method to validate the results. Thus, previous work neglected this validation step or had it done manually-by hiring qualified security specialists. In this work, we address the key-hacker identification problem including a systematic method based on reputation to validate the results. Particularly, we study how three different approaches-content, social network and seniority-based analysis-perform individually and combined to identify key-hackers on darkweb forums, aiming to confirm the following two hypotheses: 1) a hybridization of these approaches tends to produce better results when compared to the individual ones; 2) a model conceived to identify key-hackers in one forum can be generalized to other forums that lack a user reputation system or have a deficient one. We conduct our experiments using a carefully selected set of features, showing how an optimization metaheuristic obtains better performance when compared to machine learning algorithms that attempt to identify key-hackers.

IEEE TransAI, 2022

While deep neural networks have led to major advances in image recognition, language translation,... more While deep neural networks have led to major advances in image recognition, language translation, data mining, and game playing, there are well-known limits to the paradigm such as lack of explainability, difficulty of incorporating prior knowledge, and modularity. Neuro symbolic hybrid systems have recently emerged as a straightforward way to extend deep neural networks by incorporating ideas from symbolic reasoning such as computational logic. In this paper, we propose a list desirable criteria for neuro symbolic systems and examine how some of the existing approaches address these criteria. We then propose an extension to generalized annotated logic that allows for the creation of an equivalent neural architecture comprising an alternate neuro symbolic hybrid. However, unlike previous approaches that rely on continuous optimization for the training process, our framework is designed as a binarized neural network that uses discrete optimization. We provide proofs of correctness and discuss several of the challenges that must be overcome to realize this framework in an implemented system.

Advances in Argumentation in Artificial Intelligence (AI3 ), 2023

Defeasible Logic Programming (DeLP) is a structured argumentation formalism that uses a dialectic... more Defeasible Logic Programming (DeLP) is a structured argumentation formalism that uses a dialectical process to decide between contradictory conclusions. Such conclusions are supported by arguments, which are compared using a comparison criterion, to decide which one prevails in conflict situations. The definition of a formal comparison criterion is a central problem in structured argumentation, which is typically assumed to be provided by the user or knowledge engineer. In this work, we propose an integration between an argumentative approach to defeasible reasoning, such as DeLP, and machine learning models. Concretely, our goal is to train a neural network to learn a comparison criterion between arguments given a training set comprised of pairs of arguments labeled with which one prevails. We conducted several experiments, using a synthetic DeLP program generator, in order to assess the performance of a neural architecture under different kinds of DeLP programs. Our results show that under specific circumstances, a comparison criterion for arguments can be successfully learned by data-driven models.

The growing popularity of neuro symbolic reasoning has led to the adoption of various forms of di... more The growing popularity of neuro symbolic reasoning has led to the adoption of various forms of differentiable (i.e., fuzzy) first order logic. We introduce PyReason, a software framework based on generalized annotated logic that both captures the current cohort of differentiable logics and temporal extensions to support inference over finite periods of time with capabilities for open world reasoning. Further, PyReason is implemented to directly support reasoning over graphical structures (e.g., knowledge graphs, social networks, biological networks, etc.), produces fully explainable traces of inference, and includes various practical features such as type checking and a memory-efficient implementation. This paper reviews various extensions of generalized annotated logic integrated into our implementation, our modern, efficient Python-based implementation that conducts exact yet scalable deductive inference, and a suite of experiments. PyReason is available at: github.com/lab-v2/pyreason.

NeSy, 2024

Metacognition is the concept of reasoning about an agent's own internal processes and was origina... more Metacognition is the concept of reasoning about an agent's own internal processes and was originally introduced in the field of developmental psychology. In this position paper, we examine the concept of applying metacognition to artificial intelligence. We introduce a framework for understanding metacognitive artificial intelligence (AI) that we call TRAP: transparency, reasoning, adaptation, and perception. We discuss each of these aspects in-turn and explore how neurosymbolic AI (NSAI) can be leveraged to address challenges of metacognition.

IEEE ICSC, 2024

Hallucinations and reasoning errors limit the ability of large language models (LLMs) to serve as... more Hallucinations and reasoning errors limit the ability of large language models (LLMs) to serve as a natural language interface for various prompts. Meanwhile, error prediction in large language models often relies on domain-specific information. In this paper, we present domain independent measures for quantification of error in the response of a large language model based on the diversity of responses to a given promptspecifically considering components of the response. This results in an approach that is well-suited for prompts where the response can be viewed as an answer set such as semantic prompts, a common natural language interface use-case. We describe how three such measures-based on entropy, Gini impurity, and centroid distance-can be employed. We perform a suite of experiments on multiple datasets and temperature settings to demonstrate that these measures strongly correlate with the probability of failure. Additionally, we present empirical results demonstrating how these measures can be applied to few-shot prompting, chain-of-thought reasoning, and error detection.

IEEE ICSC, 2024

Recent advances in reinforcement learning (RL) have shown much promise across a variety of applic... more Recent advances in reinforcement learning (RL) have shown much promise across a variety of applications. However, issues such as scalability, explainability, and Markovian assumptions limit its applicability in certain domains. We observe that many of these shortcomings emanate from the simulator as opposed to the RL training algorithms themselves. As such, we propose a semantic proxy for simulation based on a temporal extension to annotated logic. In comparison with two high-fidelity simulators, we show up to three orders of magnitude speed-up while preserving the quality of policy learned in addition to showing the ability to model and leverage non-Markovian dynamics and instantaneous actions while providing an explainable trace describing the outcomes of the agent actions.

ICLP, 2024

In this paper, we study the problem of visual question answering (VQA) where the image and query ... more In this paper, we study the problem of visual question answering (VQA) where the image and query are represented by ASP programs that lack domain data. We provide an approach that is orthogonal and complementary to existing knowledge augmentation techniques where we abduce domain relationships of image constructs from past examples. After framing the abduction problem, we provide a baseline approach, and an implementation that significantly improves the accuracy of query answering yet requires few examples.

CIMK, 2024

Recent advances in Hierarchical Multi-label Classification (HMC), particularly neurosymbolic-base... more Recent advances in Hierarchical Multi-label Classification (HMC), particularly neurosymbolic-based approaches, have demonstrated improved consistency and accuracy by enforcing constraints on a neural model during training. However, such work assumes the existence of such constraints a-priori. In this paper, we relax this strong assumption and present an approach based on Error Detection Rules (EDR) that allow for learning explainable rules about the failure modes of machine learning models. We show that these rules are not only effective in detecting when a machine learning classifier has made an error but also can be leveraged as constraints for HMC, thereby allowing the recovery of explainable constraints even if they are not provided. We show that our approach is effective in detecting machine learning errors and recovering constraints, is noise tolerant, and can function as a source of knowledge for neurosymbolic models on multiple datasets, including a newly introduced military vehicle recognition dataset. CCS CONCEPTS • Computing methodologies → Knowledge representation and reasoning; Rule learning; Logical and relational learning.

—Human trafficking is among the most challenging law enforcement problems which demands persisten... more —Human trafficking is among the most challenging law enforcement problems which demands persistent fight against from all over the globe. In this study, we leverage readily available data from the website " Backpage " – used for classified advertisement– to discern potential patterns of human trafficking activities which manifest online and identify most likely trafficking related advertisements. Due to the lack of ground truth, we rely on two human analysts –one human trafficking victim survivor and one from law enforcement, for hand-labeling the small portion of the crawled data. We then present a semi-supervised learning approach that is trained on the available labeled and unlabeled data and evaluated on unseen data with further verification of experts.

—In this paper, we present an operational system for cyber threat intelligence gathering from var... more —In this paper, we present an operational system for cyber threat intelligence gathering from various social platforms on the Internet particularly sites on the darknet and deepnet. We focus our attention to collecting information from hacker forum discussions and marketplaces offering products and services focusing on malicious hacking. We have developed an operational system for obtaining information from these sites for the purposes of identifying emerging cyber threats. Currently, this system collects on average 305 high-quality cyber threat warnings each week. These threat warnings include information on newly developed malware and exploits that have not yet been deployed in a cyber-attack. This provides a significant service to cyber-defenders. The system is significantly augmented through the use of various data mining and machine learning techniques. With the use of machine learning models, we are able to recall 92% of products in marketplaces and 80% of discussions on forums relating to malicious hacking with high precision. We perform preliminary analysis on the data collected, demonstrating its application to aid a security expert for better threat analysis.

Each day, approximately 500 missing persons cases occur that go unsolved/unresolved in the United... more Each day, approximately 500 missing persons cases occur that go unsolved/unresolved in the United States. The non-profit organization known as the Find Me Group (FMG), led by former law enforcement professionals, is dedicated to solving or resolving these cases. This paper introduces the Missing Person Intelligence Synthesis Toolkit (MIST) which leverages a data-driven variant of geospatial abductive inference. This system takes search locations provided by a group of experts and rank-orders them based on the probability assigned to areas based on the prior performance of the experts taken as a group. We evaluate our approach compared to the current practices employed by the Find Me Group and found it significantly reduces the search area-leading to a reduction of 31 square miles over 24 cases we examined in our experiments. Currently, we are using MIST to aid the Find Me Group in an active missing person case.

—A major challenge in cyber-threat analysis is combining information from different sources to fi... more —A major challenge in cyber-threat analysis is combining information from different sources to find the person or the group responsible for the cyber-attack. It is one of the most important technical and policy challenges in cyber-security. The lack of ground truth for an individual responsible for an attack has limited previous studies. In this paper, we take a first step towards overcoming this limitation by building a dataset from the capture-the-flag event held at DEFCON, and propose an argumentation model based on a formal reasoning framework called DeLP (Defeasible Logic Programming) designed to aid an analyst in attributing a cyber-attack. We build models from latent variables to reduce the search space of culprits (attackers), and show that this reduction significantly improves the performance of classification-based approaches from 37% to 62% in identifying the attacker.

—Predicting when an individual will adopt a new behavior is an important problem in application d... more —Predicting when an individual will adopt a new behavior is an important problem in application domains such as marketing and public health. This paper examines the performance of a wide variety of social network based measurements proposed in the literature-which have not been previously compared directly. We study the probability of an individual becoming influenced based on measurements derived from neighborhood (i.e. number of influencers, personal network exposure), structural diversity, locality, temporal measures, cascade measures , and metadata. We also examine the ability to predict influence based on choice of classifier and how the ratio of positive to negative samples in both training and testing affect prediction results-further enabling practical use of these concepts for social influence applications.

—Marketplaces specializing in malicious hacking products-including malware and exploits-have rece... more —Marketplaces specializing in malicious hacking products-including malware and exploits-have recently become more prominent on the darkweb and deepweb. We scrape 17 such sites and collect information about such products in a unified database schema. Using a combination of manual labeling and unsupervised clustering, we examine a corpus of products in order to understand their various categories and how they become specialized with respect to vendor and marketplace. This initial study presents how we effectively employed unsupervised techniques to this data as well as the types of insights we gained on various categories of malicious hacking products.

— Information cascades exist in a wide variety of platforms on Internet. A very important real-wo... more — Information cascades exist in a wide variety of platforms on Internet. A very important real-world problem is to identify which information cascades can " go viral ". A system addressing this problem can be used in a variety of applications including public health, marketing and counter-terrorism. As a cascade can be considered as compound of the social network and the time series. However, in related literature where methods for solving the cascade prediction problem were proposed, the experimental settings were often limited to only a single metric for a specific problem formulation. Moreover, little attention was paid to the run time of those methods. In this paper, we first formulate the cascade prediction problem as both classification and regression. Then we compare three categories of cascade prediction methods: centrality based, feature based and point process based. We carry out the comparison through evaluation of the methods by both accuracy metrics and run time. The results show that feature based methods can outperform others in terms of prediction accuracy but suffer from heavy overhead especially for large datasets. While point process based methods can also run into issue of long run time when the model can not well adapt to the data. This paper seeks to address issues in order to allow developers of systems for social network analysis to select the most appropriate method for predicting viral information cascades.