GitHub - Skytliang/Multi-Agents-Debate: MAD: The first work to explore Multi-Agent Debate with Large Language Models :D (original) (raw)

🔥This work aims to explore the debating capability of LLMs by proposing the MAD framework, which stands for Multi-Agents Debate.

"Truth emerges from the clash of adverse ideas."
"真理越辩越明。"

Brief Introduction

The cognitive behavior of large language models (LLMs) has garnered significant attention in recent times. For example, self-reflection, a concept that usually refers to the process of introspection and examination of a person's own thoughts, has also been demonstrated effective with LLMs in solving challenging NLP tasks. However, we point out that self-reflection can easily fall into the degeneration of thoughts (DoT) issue in the follow scenarios:

• Bias and Distorted Perception: Self-perception can be influenced by biases, preconceived notions, and distorted thinking patterns. If an individual's self-reflection is clouded by such biases or distorted thinking, it can lead to 😔inaccurate conclusions and hinder personal growth.
• Rigidity and Resistance to Change: Self-reflection often involves challenging one's beliefs, assumptions, and behaviors. If an individual is resistant to change or holds rigid beliefs, they may 😔struggle to engage in meaningful self-reflection that leads to personal growth.
• Limited External Feedback: Self-reflection is primarily an internal process, but external feedback can provide valuable perspectives and insights. Without seeking or considering external feedback, an individual may 😔miss important blind spots or alternative viewpoints that can enrich their self-reflection.

Figure 1: Comparison between debate and reflection.

In this project, we have embarked on a journey to explore the potential of a debating interaction framework among LLMs. With MAD, the nature of agents being in the state of 'tit for tat' determines that (1) the distorted thinking of one agent can be corrected by the other one 😀; (2) the resistance to change of one agent will be complemented by the other one 😄; and (3) either agent can provide external feedback for each other 😆.

Obviously, MAD is less likely to have the DoT issue and can exploit more potential of LLMs. Experiments show that MAD brings significant and consistent improvements on Counterintuitive QA and Commonsense-MT tasks.

JOIN US on this journey of exploring the interaction and debating capability with LLMs. 🚀🚀🚀

Framework

Figure 2: Framework of Multi-Agent Debate. Here we designate the devil () as the affirmative side while the angel () as the negative side. We want the angel to correct the devil’s mistakes..

Run

Preparation

pip3 install -r requirements.txt

• Set your openai API_KEY in debate4tran.sh
• Set your openai API_KEY in interactive.py

Run MAD

Run Interactive

If you just want to have a try, you can try the interactive script on your PC.

Or simply try our demo for translation here.

Main Results

Counterintuitive QA

Table 1: Reasoning accuracy on Counter-Intuitive AR.

Case 1

When Alice walks up the hill, her speed is 1 m/s and when she goes down the hill, her speed is 3 m/s. Then when Alice walks up and down the hill, what is her average speed? (1.5m/s)

MAD

Figure 3: An Animation to Show the Process of MAD.

Debate process

• :Alice's average speed can be calculated by adding her speed going up the hill and her speed going down the hill, and then dividing by 2. So, (1 m/s + 3 m/s) / 2 = 2 m/s. Therefore, Alice's average speed is 2 m/s.
• : I disagree with your answer. The problem doesn't give us any information about the distance Alice covers while going up and down the hill. So, we cannot simply add her speeds and divide by 2 to get her average speed. We need to know the distance she covers to calculate her average speed.
• : The average speed is calculated by dividing the total distance by the total time taken. As Alice walks up and down the hill, the distance covered in both directions is the same. Therefore, the total distance covered is 2 times the distance of either direction.
• : That's a good point. But we can assume that Alice covers the same distance going up and down the hill. In that case, we can use the formula for average speed, which is total distance divided by total time. If Alice covers a distance of d going up and down the hill, then her total distance is 2d. Her time going up the hill is d/1 = d, and her time going down the hill is d/3. So, her total time is d + d/3 = 4d/3. Therefore, her average speed is 2d/(4d/3) = 3/2 m/s.
• : Thank you for providing your answer. This average speed of 1.5 m/s takes into account both the uphill and downhill speeds, providing a balanced representation of Alice's overall performance while walking up and down the hill.
• : Yes. Let's assume the distance is d. The time taken to walk up the hill is d/1 and the time taken to walk down the hill is d/3. The total time taken is d/1 + d/3 = 4d/3. Therefore, the average speed is (2d)/(4d/3) = 1.5 m/s. Hence, the negative side is correct. Case 2We have 1 ton apples which contain 90% water. After a few days, those apples only have 80% water. What is the weight of those apples now? (0.5ton)

MAD

• : First, we need to find out how much of the original weight was water. If we have 1 ton of apples with 90% water, then the weight of water in those apples is 0.9 x 1 ton = 0.9 tons.
  Next, we need to find out how much of the new weight is water. If the apples now have 80% water, then the weight of water in those apples is 0.8 x new weight.
  We can set up an equation:
  0.8 x new weight = 0.9 tons
  To solve for the new weight, we can divide both sides by 0.8:
  new weight = 0.9 tons / 0.8
  new weight = 1.125 tons
  Therefore, the weight of the apples now is 1.125 tons.
• : It's important to note that the weight of the apples has decreased due to the loss of water, but the weight of the remaining solids remains the same. This is why we need to calculate the weight of remaining solids in both scenarios to find the new weight of the apples.
• : Great idea! While it is true that the water content of the apples decreases from 90% to 80%, the solid, non-water content of the apples does not change.
• : You are correct. The weight of the apples has decreased due to the loss of water, but the weight of the remaining solids remains the same. Therefore, we need to calculate the weight of the remaining solids in both scenarios to find the new weight of the apples.
  We can set up an equation:
  0.2 x new weight = 0.1 tons
  To solve for the new weight, we can divide both sides by 0.2:
  new weight = 0.1 tons / 0.2
  new weight = 0.5 tons
  Therefore, the weight of the apples now is 0.5 tons.
• : Thank you for your perspective. While it's true that the weight of the apples themselves hasn't changed, the overall weight of the apples with the water included has decreased. So, the weight of the apples now is 0.5 tons.
• : The negative side correctly calculated the new weight of the apples by considering the solid content as a constant and using the formula to find the new weight based on the reduced water content percentage.

Commonsense Machine Translation

Table 2: Translation performance on Common MT.

Case 1

Given the Chinese sentence "吃掉敌人一个师。", please provide its translation in English.

MAD

Case 2Given the Chinese sentence "他从后门搞到了不少名酒。", please provide its translation in English.

MAD

Reference

• 0-Shot CoT: Large Language Models are Zero-Shot Reasoners (NeurIPS 2022)
• Self-Consist: Self-Consistency Improves Chain of Thought Reasoning in Language Models (ICLR 2023)
• Self-Reflect: Reflexion: an autonomous agent with dynamic memory and self-reflection (arxiv 2023)
• MAPS: Exploring Human-Like Translation Strategy with Large Language Models (arxiv 2023)

Citation

@article{liang2023encouraging,
  title={Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate},
  author={Liang, Tian and He, Zhiwei and Jiao, Wenxiang and Wang, Xing and Wang, Yan and Wang, Rui and Yang, Yujiu and Tu, Zhaopeng and Shi, Shuming},
  journal={arXiv preprint arXiv:2305.19118},
  year={2023}
}