causal AI (original) (raw)

What is causal AI?

Causal AI is a form of artificial intelligence (AI) designed to identify and understand the cause and effect of relationships across data.

Unlike large language models and generative AI, which are limited to recognizing and analyzing correlations across data, causal AI promises deeper explainability and reduced bias by leveraging causal inference to identify root causes in any dataset and, in turn, model accurate hypotheticals.

Causal inference, the core methodology behind causal AI, uses data to determine the independent effect of an event and draw cause-and-effect -- or causal -- conclusions. Beyond the observational data, causal AI employs techniques like causal discovery algorithms and structural causal models to learn and infer the cause-and-effect relationship of different data points, outstripping traditional machine learning (ML) capabilities. Causal AI can generate accurate responses to queries regarding the impact on a calculation if a specific variable changes.

Some of the early foundations for causal AI were defined in 2000 by Judea Pearl in research titled "Causality: Models, Reasoning and Inference," published by Cambridge University Press in 2003.

How does causal AI work?

Causal AI leverages causal inference techniques on observational data to model the dependencies and causal relations among and between different events and variables. The resulting causal models provide explainability by capturing the mechanisms that drive outcomes. Causal AI uses these models to answer hypothetical what-if questions -- known as counterfactual questions -- and estimate the effects of interventions.

From an ML perspective, causal AI relies on multiple methodologies, such as causal inference and fault tree analysis, a form of root cause analysis, to model the causal relationships between different events and variables in data.

Causal AI employs causal discovery, which analyzes patterns in data to identify relationships and construct models. These models represent the cause-and-effect dependencies between variables. Causal AI also uses structural causal models that estimate the effects of interventions by modeling hypotheticals and counterfactuals.

For example, causal AI applies fault tree analysis, which utilizes Boolean logic and a top-down approach, to identify the sequence of events that caused a system failure. The process starts with the system failure event and then scrutinizes preceding events to find the root causes. The fault tree maps the relationships between component failures and overall system failures.

Causal AI models sometimes incorporate domain expertise, combining data-driven modeling with human knowledge to uncover precise causal mechanisms behind patterns observed in data.

Here is a step-by-step explanation of causal AI in action:

1. Collect observational data

Causal AI systems start by collecting large amounts of observational data that capture events, behaviors and metrics over time. This data serves as the input to uncover causal relationships.

2. Discover causal relationships

Algorithms analyze the patterns and connections in the observational data to detect potential causal relationships between variables. Techniques like causal discovery identify these connections to, in turn, construct a causal model.

3. Build causal models

These causal relationships become the framework for causal models, such as Bayesian networks or structural causal models. These models represent the causal dependencies and relationships between variables based on the discovered patterns.

4. Incorporate domain expertise

Domain experts provide additional input to the causal models by constraining or specifying known causal relationships, combining data-driven modeling with human experience and skill.

5. Estimate causal effects

The causal models use techniques like counterfactual analysis to estimate the causal effects of hypothetical interventions. Follow-up questions determine the impact of changing variables.

6. Test interventions

Organizations use causal models to test would-be interventions on a small scale or in simulated environments to predict effectiveness before broader implementation.

7. Iterate

As new data comes in, iterated causal models are refined over time to improve accuracy and value, providing ongoing explainability.

Causal AI's development tools

Causal AI techniques include causal discovery algorithms, structural causal models and counterfactual analysis. Multiple commercial tools and open source libraries support the development of causal AI.

The open source projects listed below are among the most popular causal AI projects on the GitHub code repository. Listed vendors were found with extensive web research and have a clear focus on providing commercial tools for causal AI.

• CausalML: An open source library with development led by Uber that provides causal inference capabilities.
• CausalImpact: Google-led open source effort written in the R programming language for time series causal inference.
• Causal-learn: An open source causal discovery library.
• CausaLens: Vendor that provides commercial tools and services for causal AI.
• Causely: Commercial vendor providing tools.
• Causica: Microsoft-backed open source effort providing tools for both causal discovery and inference.
• DoWhy: An open source causal inference library written in Python.

How causal AI handles missing data

Causal AI's reliability requires its models to have data that's as accurate and complete as possible. Causal AI manages the challenge of missing data with the following approaches:

• Data imputation. Data imputation is a data science technique used in different ML models to input missing values. There are numerous data imputation techniques, including K Nearest Neighbor, Missing Value Prediction, Most Frequent Value and Moving Average. Each provides a mathematical approach to determine missing values.
• Multiple imputation. Multiple imputation, an advanced form of data imputation, combines many imputed data sets to create a more complete data set.
• Inverse probability weighting (IPW). IPW is a statistical method that adjusts the weights of the existing data in an ML model to account for its missing data.

How causal AI differs from and improves on other AI

Causal AI digs deeper than other AI in search of cause-and-effect relationships among data. Here are a few of its strengths:

Main differences between causal AI and other forms of AI.

Depending on the use case, causal AI delivers specific advantages and benefits over other forms of AI, such as the following:

• Deeper explainability by identifying root causes behind outcomes.
• Reduced bias in ML data sets by revealing causal relationships rather than superficial correlations.
• Hypothetical question answeringby data scientists -- such as estimating the effect of interventions -- to gauge effectiveness prior to real-world implementation.
• Improved optimization by moving beyond predictive analytics to power new applications such as root cause analysis and scenario planning.

Real-world applications of causal AI

Causal AI isn't just a hypothetical technology. It has applications across multiple industry verticals.

Causal AI applications seek to understand the reasons behind customer churn to improve retention and identify causes of transaction decline to boost conversions.

• Healthcare. Learn the causal effects of treatments to determine optimal interventions for patients.
• Finance. Analyze the root causes behind investment risks to mitigate loss.
• Fraud detection. Identify the chain of events and conditions that enable fraud. This supports developing targeted fraud prevention measures.
• Manufacturing. Perform root cause analysis on production line failures or optimize supply chain operations.
• Government. Evaluate the impact of potential policies through simulations. Quantifying the effects of different interventions encourages evidence-based policymaking.

Causal AI's business effects

There are numerous ways businesses can leverage causal AI's benefits and applications.

• Bias removal. Since it uncovers true causal relationships, causal AI reduces bias produced from bogus correlations that might exist in data.
• Customer journey. Causal AI scrutinizes the customer journey to understand the root causes behind poor conversion rates or customer abandonment at different touchpoints. It provides explainability to help businesses optimize consumer journeys.
• Churn prevention. By identifying the factors driving customer churn, causal AI models simulate interventions to determine optimal strategies for improving retention.
• Human resources. Causal AI analyzes the factors leading to outcomes like employee turnover. It provides explainable recommendations to improve retention.
• Investment risk analysis. Causal AI helps analyze the causal mechanisms behind investment risks to develop mitigation strategies and seize new opportunities.
• Optimizing business operations. By simulating changes to pricing, promotions and marketing strategies, businesses leverage causal AI to estimate impacts on metrics and improve strategies.
• Understanding root causes. Causal AI provides explainability by revealing the underlying root causes of business events and outcomes.

What is the future of causal AI?

Though generative AI has many positive attributes, its drawbacks include AI hallucination and the inability to predict causal relationships.

As organizations continue to integrate AI into operations, the capabilities of causal AI to better understand root causes and model potential scenarios will drive its further growth and adoption. Indeed, a few months prior to ChatGPT's debut, Gartner identified causal AI as a key emerging technology.