EscapeCraft: A 3D Room Escape Environment for Benchmarking Complex Multimodal Reasoning Ability (original) (raw)

Ziyue Wang1 ♠111Equal contribution, ♠ Project lead, 🖂 Corresponding author,Yurui Dong3111Equal contribution, ♠ Project lead, 🖂 Corresponding author, Fuwen Luo1, Minyuan Ruan1, Zhili Cheng1,
Chi Chen1, Peng Li2 🖂, Yang Liu1,2 🖂
1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China
2Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
3School of Management, Fudan University, Shanghai, China

Abstract

The rapid advancing of Multimodal Large Language Models (MLLMs) has spurred interest in complex multimodal reasoning tasks in the real-world and virtual environment, which require coordinating multiple abilities, including visual perception, visual reasoning, spatial awareness, and target deduction. However, existing evaluations primarily assess the final task completion, often degrading assessments to isolated abilities such as visual grounding and visual question answering. Less attention is given to comprehensively and quantitatively analyzing reasoning process in multimodal environments, which is crucial for understanding model behaviors and underlying reasoning mechanisms beyond merely task success. To address this, we introduce MM-Escape, an extensible benchmark for investigating multimodal reasoning, inspired by real-world escape games. MM-Escape emphasizes intermediate model behaviors alongside final task completion. To achieve this, we develop EscapeCraft, a customizable and open environment that enables models to engage in free-form exploration for assessing multimodal reasoning. Extensive experiments show that MLLMs, regardless of scale, can successfully complete the simplest room escape tasks, with some exhibiting human-like exploration strategies. Yet, performance dramatically drops as task difficulty increases. Moreover, we observe that performance bottlenecks vary across models, revealing distinct failure modes and limitations in their multimodal reasoning abilities, such as repetitive trajectories without adaptive exploration, getting stuck in corners due to poor visual spatial awareness, and ineffective use of acquired props, such as the key. We hope our work sheds light on new challenges in multimodal reasoning, and uncovers potential improvements in MLLMs capabilities.111GitHub repo: https://github.com/THUNLP-MT/EscapeCraft. 222Home page: https://thunlp-mt.github.io/EscapeCraft.

Figure 1: Illustration of our proposed room escape environment EscapeCraft, which allows us to generate customized room scenes (left) and define groundtruth reasoning path of tasks (right). Based on EscapeCraft, we create MM-Escape benchmark, targeting at evaluating both the task completion performance and the entire multimodal reasoning process of MLLMs.

1 Introduction

The rapid development of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have driven the advancement of diverse multimodal systems and applications for academic research [43, 2], industrial engineering [16], and everyday assistance [35, 14]. Multimodal reasoning is essential for these applications that require integrating multiple abilities such as visual perception, spatial awareness, and visual grounding [36]. For example, it enhances autonomous driving by improving the holistic understanding of multi-view information and localization, which are essential for vehicle actions and planning [11, 13], and also advances the general-purpose assistants in better performing visual and multimodal tasks in the wild [18, 22].

Despite significant attention and effort towards improving multimodal reasoning abilities of MLLMs [36, 23, 48], comprehensive evaluation remains underexplored for two key reasons. First, fundamental tasks such as visual grounding [49, 47, 9] and image captioning [26, 1] are conducted in constraint environment and straightforward objectives, reducing the need for autonomous exploration. They primarily focus on identifying correct answers, such as bounding boxes and objects, without requiring coordinating multiple multimodal abilities. Second, while multimodal tasks in open-world settings [27, 21, 30] involve complex environments and objectives, they emphasize final task completion, often measured by success rate [19]. This results in a lack of profound analysis over the reasoning process, leading to potentially inaccurate assessments of multimodal reasoning capabilities. Moreover, some open-world tasks provide structured knowledge libraries [10, 37] that standardize the reasoning mechanisms. These consequently limit the autonomy of models to conduct multimodal reasoning, making the reasoning more reliant on predefined knowledge rather than exploration of multimodal surroundings.

We argue that in open multimodal environment, including real-world settings and virtual simulators, complex multimodal reasoning should not be solely assessed by task completion results or isolated tasks. Instead, it is more practical and realistic to examine how models autonomously coordinate across multiple multimodal reasoning abilities. Recently, open environments such as Habitat [31], AI2-THOR[17], and OsWorld[41] are widely discussed, where MLLMs are required to exhibit complex reasoning skills including visual searching, spatial understanding, tool utilization, and long-term decision-making. However, as summarize in Table 1, there is a constraint on tasks or environments especially designed for evaluating multimodal reasoning in open-ended and interactive settings, leaving the evaluation of complex multimodal reasoning underestimated.

In this paper, we introduce MM-Escape, an extensible benchmark for evaluating complex multimodal reasoning, inspired by real-world escape games. To achieve this, we develop EscapeCraft, a customizable open environment that engages models in free-form exploration through the room escape task, assisting in comprehensive assessments of their multimodal reasoning abilities. MM-Escape provides solutions to the above issues of excessive concerns on task completion and simplified objectives, by emphasizing intermediate behaviors alongside final task completion. It measures the entire reasoning process consists of compound abilities. While our designed games are intuitive for human players, they pose significant challenges for models regarding reasoning paths and interaction behaviors, exposing key limitations in current multimodal reasoning capabilities. As an additional bonus, we introduce post-game debriefing task, which is feasible only for the most challenging settings. It requires models to recall the escape process and reconstruct the story via collected clues, which assist in the future assessment of more complicated reasoning tasks.

We conduct comprehensive experiments and derive two key findings. First, although the overall performance remains far from human-level, recent MLLMs demonstrate strong multimodal reasoning ability. Notably, models like GPT-4o and Gemini-1.5-Pro achieve high escape rates and exhibit human-like exploration and reasoning strategies. While their interaction success rates are halved with steps more than doubled compared to human results. Second, as game difficulty increases, the model performance sharply declines, revealing distinct failure modes across models. For instance, GPT-4o tends to repeat actions and get trapped in historical trajectories when reasoning paths grow longer, Gemini struggles with spatial awareness and often gets stuck in corners. We also thoroughly discuss post-game debriefing and other extensible settings in Section 5. These findings highlight the need for greater attentions toward comprehensive analysis and improvement of multimodal reasoning abilities. Our contributions are as follows:

• •
  We introduce a benchmark, MM-Escape, to advance comprehensive evaluation of multimodal reasoning for MLLMs, by quantitatively evaluating intermediate reasoning process alongside task completion performance.
• •
  Our benchmark features free-form exploration, requiring models to autonomously coordinate multiple multimodal reasoning abilities in the multimodal room escape task.
• •
  We thoroughly investigate model behaviors using MM-Escape and identify distinct limitations across models. Our analysis provides detailed insights, highlighting future optimization and potential real-world applications.

Table 1: Benchmarks aiming at evaluating complex reasoning ability of models. URP means unpredictable reasoning path. PA means process assessment.

2 Related Work

2.1 Complex Reasoning Abilities of MLLMs

Recent research on MLLMs has moved beyond addressing simple tasks, such as image captioning and image retrieval, and instead focuses on enhancing model abilities towards more complex tasks and more realistic scenarios. For example, MLLMs can operate digital devices such as computers [44, 28] and mobile phones [34, 40], play video games requiring long action sequences [3, 7], and perform robotic manipulation in the real world [12, 50]. Unlike tasks such as visual question answering and visual grounding, which have standardized answers, these complex tasks allow diverse solutions and engage in open environments. As a result, multimodal reasoning ability is becoming increasingly important for achieving more advanced targets.

2.2 Complex Multimodal Reasoning Evaluation

Researchers have made efforts to evaluate the visual reasoning ability of MLLMs in complex tasks which require models to coordinate multiple multimodal abilities [25, 24, 45]. However, many of them leverage videos with predefined perspectives and trajectories as input. Models cannot interact with environments by themselves.

Some other works focus on complex open worlds, such as Minecraft [33, 38, 30, 39, 21, 20], to evaluate model abilities of multimodal reasoning. However, two issues still exist. First, due to clearly defined game rules, standardized guides are available from websites like Minecraft Wiki 333https://minecraft.wiki/, and training datasets of game demonstrations [15], making it feasible for models to reason based on their commonsense or relying on external knowledge rather than observations during free-form exploration. Second, evaluations are often outcome-oriented, for example, focusing on specific goals like obtaining a diamond in Minecraft, while how models explore the open world is ignored. Similar issues exist in other complex open-ended tasks in open worlds like computer use [41] and robotic manipulation [32, 46].

Compared to existing benchmarks, our work features in three aspects: a) we support sufficient interactions between models and environments; b) environments can be flexibly configured, preventing models from solely relying external knowledge to achieve the goals; c) reasoning processes are also evaluated alongside final results. These enable us to understand model abilities more comprehensively.

2.3 Multimodal v.s. Pure Text Environments

Previous works investigated reasoning ability of models in pure text scenarios by allowing models to interact with environments via text interfaces [6, 42, 29]. However, some multimodal information cannot be easily and precisely expressed in verbal forms, hindering reasoning ability based on multimodal information to be evaluated in pure text environments. Additionally, models enjoy enhanced decision-making autonomy within multimodal environments compared to text environments. For example, models may decide their rotation angles or travel distances based on multimodal inputs when requiring to observe target objects or nearby surroundings, but it is hard to make such decisions based on text descriptions. Therefore, we believe that multimodal environments are essential for evaluating reasoning abilities of models, and propose EscapeCraft and MM-Escape to address the above issues.

Figure 2: An example of success escape path from Gemini-1.5-pro. For the first four steps, it rotates, without moving, to observe the surroundings, and then starts moving towards the direct on the observed tables in the following steps. For step 5 and 6, we zoom in to the key to make it clearer. The key is distorted because the view of model is tilting downwards to observe objects on the table. From step 10 to 14, the model is already close to the door, and is slowly rotating and tilting to locate the door.

3 MM-Escape

In this paper, we address comprehensive investigation of complex multimodal reasoning for MLLMs by introducing MM-Escape, an extensible benchmark. To support this, we design EscapeCraft, a highly customizable and open environment that engages models in free-form exploration for quantitatively assessing the intermediate multimodal reasoning process, as depicted in Figure 1. We highlight that complex multimodal reasoning requires an autonomous integration of multiple multimodal abilities, and that evaluating the reasoning process is as crucial as assessing final task completion. We will first describe our introduced tasks in the following section, and detail the construction of our benchmark along with its corresponding environment.

3.1 Task Definition

Inspired by real-world game, room escape, we propose to place MLLMs in an escape escape scenario to investigate their multimodal reasoning abilities in an open-environment, where models should autonomously explore multimodal surroundings, and reason toward both temporary and long-term goals. We name the introduced task, multimodal room escape. It consists of a mandatory target, the room escaping task, aiming at escaping a locked room, and an optional target the post-game debriefing task, requiring to reconstruct the story discovered during the escaping. Our multimodal room escape task assesses the entire reasoning process rather than solely focusing on final game completion.

Room Escaping Task.

This task presents a ultimate goal of exiting the room, as shown in Figure 1 (Game Target) and a detailed example is demonstrated in Figure 2. It requires models to fully explore and interact with the multimodal environment, search for props and clues, identify the exit, and correctly use props to unlock the door. We do not exploit step-by-step instructions to restricted model actions, ensuring them to freely explore the environment without constraints, and automatically deduce current or short-term goals. This task evaluates the integrated multimodal reasoning ability, including object recognition, visual search, visual reasoning, target deduction, spatial reasoning, and prop utilization. Please refer to Supplementary Materials Sec. J for detailed discussion over full required abilities.

Post-game debriefing task differs from the room escaping task that involves reasoning about past experiences, current states, and future plans, as it takes place after the game is completed444This task is also common in real-world escape game.. It serves as an optional tasks because its preliminary requirement is to collect all props in the game and exit the room successfully, which is quite challenging for most of current models. Therefore, for a fair comparison, we only apply this target to cases where models correctly complete the game as shown in Figure 1. It emphasizes logicality and consistency of reconstruction, requiring models to reflect on the past experiences and events.

3.2 Construction and Design of Environment

We develop EscapeCraft to place models in an escape game, a free-form exploration environment, to comprehensively investigate complex multimodal reasoning ability of models. It supports customizable and extensible scene generation for our escape game with minimal manual effort.

Room Scene Generation

To support room escape task, we generate diverse and interactable room environments, which requires efficient and large-scale scene data generation and rendering. We develop an environment, EscapeCraft, by extending ProcTHOR [8] and Legent [4], which are originally labor-intensive regarding the scene generation process. We enable automatic size adaptation to predefined or customized configurations, such as room scale, number of rooms, and required furniture, by incorporating 3D furniture models with annotated size information. Critical objects are made fully interactable to serve as props and clues to assist in completing the task. The flexibility and interactivity of objects allow for automatic large-scale 3D room generation following the requirement of our benchmark. EscapeCraft is highly customizable and extensible, making it well-suited for the room escape task. Detailed construction can be found in Supplementary Material C.1.

Action Space

We define three types of actions, moving, view adjustment, and interaction. The moving action, i.e. moving forward, allows the model to change its position to perceive objects at different depth. View adjustment enables perception from different angles and facilitates object selection for interaction, including horizontal or vertical rotation, and looking at specific coordinates. Interaction actions contain grabbing, using, reading, and inputting, allowing models to obtain and utilize props from the environment, and process messages displayed by the props. These actions can be executed individually or integratedly in a multi-action way. See Supplementary Materials Sec. C.1.2 for details.

Inventory system

To enable model players to acquire and utilize props within the environment, we design an inventory system that allows models to store and manage acquired items, access detailed information about them, and use them as needed. This system assists models in effectively using props, and successfully escaping the room.

3.3 MM-Escape Benchmark

Figure 3: Illustration of difficulties, showing required paths for the reasoning process of successful escapes regarding each levels. The levels can be customized and extended as depicted by “difficulty-n” example, and as demonstrated in Figure 1 (lower right part).

Game Settings.

We introduce an automatic reasoning chain generation procedure by configuring the Prop Chain. In detail, this is a singly linked list representing the ordered sequence of items and interactions required to complete the game. Each node in the chain corresponds to an interactive element, such as a key, a locked box, or a note with a password, where the tail node represents the exit point of the game. To construct a complete escape game setting, we annotate the links between nodes in the prop chain to define the ways to obtain different props (such as unlimited acquisition or requiring a key to open, etc.) and their inclusion relationships (for example, a key can be placed in a box). See Supplementary Material Sec. C.2 for details.

Following this, we employ varying difficulty levels in MM-Escape to facilitate in-depth assessments of complex multimodal reasoning. Shown in Figure 3, difficulty is primarily determined by the predefined prop chain of a game, where longer chains correspond to higher difficulty. We define three standard difficulty levels for individual rooms:

• •
  Difficulty-1: The simplest one-hop reasoning path where no props are needed to unlock the door. Models can exit by locating the door and interacting with it directly.
• •
  Difficulty-2: A two-hop reasoning path requiring an additional key or password compared to Difficulty-1. Models should search for the key or password and interact with it to unlock the door.
• •
  Difficulty-3: A three-hop reasoning path requiring both a password and a key, with one additional hop to Difficulty-2. This level challenges models with spatial reasoning, visual search, and prop utilization.

Since the prop chain can grow infinitely, our difficulty levels are inherently extendable. Moerover, the type of questions or tasks in each reasoning hop are customizable and interchangeable, further enhancing the difficulty and flexibility of MM-Escape. We also explore some extended settings that incorporate with other tasks, such as embodied QA and visual logical reasoning, with case studies in Section 5.2

Additionally, to further investigate the behavior and complex multimodal reasoning abilities of MLLMs, we introduce a multi-room setting by combining two standard single rooms. We create multiple multi-room combinations, each containing two individual rooms. The configurations include: two Difficulty-1 rooms, two Difficulty-2 rooms, and a mixed setting of Difficulty-1 and Difficulty-2. In the multi-room setting, models start in the first room, which has only one exit. Upon successfully exiting, models enter the second room and search for the final game exit. However, this setting presents a greater challenge than the single-room scenario, as there are two doors in the second room, requiring models to distinguish between exit and entrance based on their corresponding surroundings.

Statistics of MM-Escape

For individual room settings, we generated 11 scenes for each of Difficulty-1 and Difficulty-2, and 21 scenes for Difficulty-3. As there are two types of props required by Difficulty-3, we enable a key-first and a password-first prop chains. For multi-room settings, we generated 10 scenes for the three different combinations introduced above. There are totally 63 scenes for standard evaluation of our benchmark, which could be further extended for future research without adapting the environment. These scenes are categorized into four distinct styles: living room (14), kitchen (19), bathroom (19), and bedroom (11). And different objects are automatically placed within each scene to correspond to its specific style. On average, difficulty-1 scenes contain 20.18 objects, difficulty-2 contains 14.55, and difficulty-3 contains 15.24. The objects are logically arranged in a manner consistent with real-life settings and randomly distributed within the rooms to serve as part of the tasks for models.

Table 2: Results of standard single room setting. Prop: Prop Gain; Steps: average steps used to complete the game; Grab SR: the precision of grabbing; Grab Ratio: the portion of grabbing actions regarding the total consumed steps. Note that Difficulty-1 requires no prop, and the prop gain is therefore omitted for this setting. The max allowed steps are 50, 75, 100 for Difficulty-1, -2, -3 respectively. The best score of each metrics is bolded and the second is underlined.

Table 3: Performance on multi-room setting for different room scenes. To assist in the more challenging setting, we provide models with a full successful escape path from Room 1 (9 steps) for self-reflection when they try to unlock Room 2. Hence, the Prop Gain (Prop (%)) in the results refers only to Room 2. Further challenges of escaping from the very beginning of multi-room setting are discussed in Supplementary Material F.

Metrics of MM-Escape

To comprehensively evaluate complex multimodal reasoning ability, we propose a set of metrics for the intermediate process, alongside an indicator for final task completion. For the room escape task, we use average escape rate (ER) as the indicator of game completion, and design five metrics for measuring intermediate interactions, including prop gain, average steps, grab count, grab success rate, and grab ratio. Denoting the total steps as S𝑆Sitalic_S, amount of succeeded grabbing action as Ng⁢r⁢a⁢bT⁢Psuperscriptsubscript𝑁𝑔𝑟𝑎𝑏𝑇𝑃N_{grab}^{TP}italic_N start_POSTSUBSCRIPT italic_g italic_r italic_a italic_b end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T italic_P end_POSTSUPERSCRIPT, grab success rate as GSR, grab ratio as Rg⁢r⁢a⁢bsubscriptR𝑔𝑟𝑎𝑏\text{R}_{grab}R start_POSTSUBSCRIPT italic_g italic_r italic_a italic_b end_POSTSUBSCRIPT we have,

The debriefing task is only applied to cases where models successfully escaped as a bonus, to further investigate the multimodal understanding and reasoning process. Because models that successfully complete the game tend to achieve high scores regarding metrics mentioned above. We employ large language models as evaluator for this task, assessing the consistency ([0,1]01[0,1][ 0 , 1 ]) of reconstructed stories with the groundtruth stories.

4 Experiments

Evaluation Setups.

We investigate both open-source models and propriety models of different scales, and evaluate their multimodal reasoning ability in level-wise approaches as outlined in Section 3.3. Specifically, we employ both single room and multi-room settings. The multi-room setting can be viewed as combinations of two single rooms, with the second room featuring two doors—one for entry and one for exit. For robust comparison, we set the temperature to 0 to eliminate token decoding diversity. The prompt templates used for evaluation are detailed in Supplementary Material Sec. K. Since the game can grow infinitely, we limit the maximum steps for each difficulties as 50, 75, and 100 respectively, for quantitative measurements. We slightly increase the max step to 80 for the most challenging multi-room setting. Since the proposed room escape task requires models to fully utilize past trajectories to gain a comprehensive understanding of the room environment, we investigate models supporting long input contexts, at least 128k, as shown in Table 2. Proprietary models include, GPT-4o555https://platform.openai.com/docs/models/gpt-4o, Gemini-1.5-pro666https://ai.google.dev/gemini-api/docs/models#gemini-1.5-pro, Claude 3.5 Sonnet777https://www.anthropic.com/news/claude-3-5-sonnet and Doubao 1.5 Pro888https://www.volcengine.com/docs/82379/1330310#doubao-1-5-vision, open-source models include Phi-3-vision, Llama-3.2-11b and Qwen2.5-VL.

Figure 4: Analysis on the grabbing behaviors with respect to the escape rate and the visibility of the exit at initial location.

Main Results of Game Completion

Results of standard single room setting, along with manual evaluation are reported in Table 2. See Supplementary Material Sec. B for details of human evaluation.

The results indicate that model performances falls significantly behind human capabilities across all metrics, highlighting the remarkable gap between current multimodal reasoning abilities and human-level capabilities. Among the models, GPT-4o demonstrates the strongest overall performance across all difficulty levels, achieving the highest average escape rate (81.36%), followed by Gemini-1.5-pro (61.06%). Other models limited success in task completion, except for Phi-3-vision-128k, which fails considering the task completion. Claude 3.5 Sonnet and Llama-3.2-11b-vision achieve moderate ER, but their performance drops significantly in Difficulty-2 and -3, suggesting weaknesses in multi-hop multimodal reasoning.

For the intermediate process, GPT-4o efficiently completes tasks with relatively fewer steps while maintaining a high grabbing success rate. Notably, Gemini-1.5-pro achieves a lower escape rate (ER) in Difficulty-2 compared to GPT-4o, yet it presents the highest Prop Gain, suggesting a strong visual search ability. For models do not survive the task completion, we can still measure their multimodal reasoning ability in terms of Prop Gain, GSR and Grab Ratio. Llama-3.2-11b-vision and Qwen-VL-Max receive identical Prop Gain, while the later presents higher GSR with a lower Grab Ratio. This implies that Qwen-VL-Max can precisely identify scenes with useful props, while Llama-3.2 is likely to adopt a greedy policy to increase grab actions so as to find more props, which is often unintentionally.

For the more challenging multi-room setting, we can derive similar conclusions. Additionally, we notice that by providing a successful path of the first room helps the model to better conduct multimodal reasoning in our task for most models, but the improvements present in different forms. For example, GPT-4o achieves an ER of 90% for Difficulty-2 when bootstrapped by a first room of Difficulty-1, and prop gain of Gemini and Llama 3.2 is improved in Difficulty-2 & -2 combination compared with the setting of a single room of Difficulty-2.

Table 4: Steps taken to get key props and to open the door, denoted as step. And the ratio of cost of each corresponding stages, denoted as cost, which is the proportion of corresponding steps out of the total steps. #PW: step/cost to obtain passwords from the beginning. #Key: step/cost to obtain keys from the beginning (Difficulty-2), and to obtain keys after the acquisition of password (Difficulty-3). #Exit: step/cost to escape after the acquisition of the last key prop.

5 Analysis and Discussions

Takeaway Observations

• •
  Distinct Human-Like Behavioral Patterns: Models exhibit unique behaviors in room escape task. Gemini tends to remain in a fixed location, typically the starting point, scanning its surroundings before taking action. In contrast, GPT-4o first observes a wider range for a global understanding of the environment.
• •
  Diverse Observation Strategies: Some models deviate from a conventional front-facing view. Gemini frequently employs a downward-facing perspective to inspect objects like tables and chairs, whereas GPT-4o predominantly relies on a front-facing view.
• •
  Common failure modes: We observe some common reasons leading to unsuccessful escape, such as movement failures and interaction challenges. Movement failures vary across models, including repetitive trajectories (GPT-4o) and getting trapped (Gemini and Claude). Interaction challenges include action combination failure (Phi-3 and Qwen-VL) and imprecise object specifying.
• •
  Adherence to Long-Term Goals: Most models remain goal-oriented, focusing on locating the exit and clues or props that could assist in escaping. However, Phi-3 consistently fails to generate valid object interactions and does not recognize doors, unlike other models, which engage with doors upon detection.

Please refer to Supplementary Material Sec. A for full version of overall observations. In the following of this section, we will further discuss model performance concerning the escape process, including the escaping path, the consistency of sub-goals, and post-game debriefing, and also provide cases to demonstrate the extensibility of our EscapeCraft.

5.1 Analysis of Entire Path

Figure 5: The extended scenario where the required password is displayed via a numerical pattern on the wall, rather than explicitly written on notes. GPT-4o completes reading it at once and exits within five steps, while Gemini struggles to repetitively search the room. Moving the pattern away from the door further challenges GPT-4o, leading to a failure of escaping.

We investigate three three key questions in this section: Q1) How many steps required to obtain props? Q2) How many steps are needed to exit the room after acquiring the core prop (key or password)? Q3) What is the relationship between grab success rate and escape outcome for each test?

For Q1, GPT-4o presents a significant advantage in step counts required to obtain the key followed by Gemini as shown in Table 4. While Claude requires fewer steps to find props in Difficulty-3, this comes at the cost of a significant decrease in escape rate. The superior performance in locating and obtaining the core props can be attributed to better understanding of task objectives and the holistic environment, and its enhanced reasoning abilities in this context. For Q2, Gemini can locate and acquire the key at a lower cost in difficulty-2, but GPT-4o outperforms in Difficulty-3, which is more complex. GPT-4o benefits from its prior memory and understanding of the room environment, gained in the process of obtaining key props, which aids it to efficiently locate the exit and escape with fewer steps compared to other models. For Q3, escape success is positively correlated with grab success rate (GSR), as shown in Figure 4(a). Higher GSR implies that models have experienced more successful interactions with the environment, potentially indicating a clearer understanding of the overall environment and ultimate goals for our task. While GSR declines with difficulty, the scores of GPT-4o and Claude 3.5 remain relatively stable compared to others, with less variation in grabbing behavior and GSR across difficulties. The low GSR of Qwen in difficulty-2 and -3 can be partly caused by the ineffective perception of the environment, inferior reasoning and interacting decision in this complex tasks, while the low GSR of Llama 3.2 is limited by its input registration of only one image at a time.

Please refer to Supplementary Materials Sec. E for detailed discussion, and Sec. D for additional examination about the moving distance and arrangement of the room.

5.2 The Extensibility of EscapeCraft

We provide an extended case study in this section, and also discuss the fully autonomous version of multi-room setting in Supplementary Materials Sec. F, and an additional customizations of escaping path in Supplementary Materials G.

We introduced an extended scenario where the required password is displayed via a numerical pattern on the wall, rather than explicitly written on notes, as shown in Figure 5. Models should recognize the pattern on the wall (password) and infer its relevance to the door. When pattern appears near the door, GPT-4o quickly identifies it and exit in the following five steps, while Gemini, despite seeing the pattern, failed to recognize it as the password and instead searched the room exhaustively repeatedly.

We further move the pattern away from the location of the door, and observe unchanged behaviors of Gemini. However, GPT-4o performs differently, by repeatedly moving between the bed and the wall without recognizing the password. It also failed to interact with the door until the game stops by the max allowance, revealing limitations in its long-term reasoning and spatial reasoning.

5.3 Analysis of Intermediate Consistency

Table 5: Consistency compared with GSR on Difficulty-3.

During experiments, we observe that some successful interactions are completed by accident instead of a proper reasoning process. For example, a model aims at a TV but obtaining a key instead. To further analyze the reasoning process concerning successful sub-goals (i.e. successful grabbing), we introduce an additional metric, intent-outcome consistency, denoted by CI⁢Osubscript𝐶𝐼𝑂C_{IO}italic_C start_POSTSUBSCRIPT italic_I italic_O end_POSTSUBSCRIPT, ranging [0,1]. This metric quantitatively assesses the intermediate process of achieving sub-goals. Specifically, it measures if model generated rationale (intent) aligns with actual environment feedback (outcome). We use GPT-4o for auto-evaluation with prompt template provided in Supplementary Materials K, where score 0 implies accidental success, and 1 reflects true valid reasoning. We calculate scores of Difficulty-3, as it involves more steps which can better reflect the intermediate consistency, and report some of the results in Table 5. GPT-4o achieves only 26.51% consistency, despite a 31.36% of GSR, implying many completed sub-goals are accidental. And so do other models, suggesting common accidental completion of sub-goals, indicating a need to improve the consistency between reasoning and perception capabilities.

5.4 Analysis on Post-game Debriefing

The post-game debriefing task requires models to recall their escape process and obtained clues, and reconstruct the whole stories. As successful escape is necessary for post-game debriefing, we only evaluate models with high success rate, that is, GPT-4o and Gemini-1.5-pro. Results show that both models fall short of ability of retelling the stories. Models pay strong attention to the processes which are directly related to the completion of room escaping, such as password acquisition. Meanwhile, they ignore background stories which are less important but also helpful for escaping. For the reason of limited model abilities, this may be a effective strategy to complete tasks. However, with the enhancement of model abilities in the future, it is necessary to improve model ability of memorization of background information. For the experiment results, please refer to Supplementary Material Sec. I.

6 Conclusions

In this paper, we introduce MM-Escape, an extensible benchmark for investigating multimodal reasoning, inspired by real-world escape games. We also develop EscapeCraft that enables models to engage in free-form exploration for assessing multimodal reasoning, to construction our benchmark. We find that MLLMs can successfully complete the simplest level of tasks, and some models even exhibiting human-like behaviors and strategies. However, performance dramatically drops as task difficulty increases, while human testers consistently succeed. MM-Escape reveals distinct failure modes across models, such as repetitive trajectories without adaptive exploration, trapped in corners without good spatial awareness. We hope our work sheds light on new challenges, and uncovers potential improvements for MLLMs.

Contributions

Ziyue Wang: Design of the escape process and post-game debriefing, implementation of EscapeCraft, all reported experiments. Paper writing: all sections and figures.

Yurui Dong: Design of the escape process and post-game debriefing, construction of 3D environment, all engineering and coding works. Paper writing: method sections and appendix sections, figures of case study.

Fuwen Luo: Design of the escape process, design and experiments of post-game debriefing. Paper writing: related works, post-game debriefing experiments and analysis.

Minyuan Ruan: Implementation of EscapeCraft, scene generation, construction of homepage. Paper writing: analysis, human evaluations, figures of analysis.

Zhili Cheng: Construction of 3D environment, design of room escape process.

Chi Chen: Design of room escape process and post-game debriefing, support on experiments.

Peng Li: Project supervision, advising of all designs, engineering, experiments, and paper writing.

Yang Liu: Project supervision, advising of all designs, engineering, experiments, and paper writing. 0000

References

• Bai and An [2018] Shuang Bai and Shan An. A survey on automatic image caption generation. Neurocomputing, 311:291–304, 2018.
• Bhattacharya et al. [2024] Manojit Bhattacharya, Soumen Pal, Srijan Chatterjee, Sang-Soo Lee, and Chiranjib Chakraborty. Large language model to multimodal large language model: A journey to shape the biological macromolecules to biological sciences and medicine. Molecular Therapy-Nucleic Acids, 35(3), 2024.
• Chen et al. [2024] Peng Chen, Pi Bu, Jun Song, Yuan Gao, and Bo Zheng. Can vlms play action role-playing games? take black myth wukong as a study case, 2024.
• Cheng et al. [2024] Zhili Cheng, Zhitong Wang, Jinyi Hu, Shengding Hu, An Liu, Yuge Tu, Pengkai Li, Lei Shi, Zhiyuan Liu, and Maosong Sun. Legent: Open platform for embodied agents. arXiv preprint arXiv:2404.18243, 2024.
• Cheng et al. [2025] Zhili Cheng, Yuge Tu, Ran Li, Shiqi Dai, Jinyi Hu, Shengding Hu, Jiahao Li, Yang Shi, Tianyu Yu, Weize Chen, et al. Embodiedeval: Evaluate multimodal llms as embodied agents. arXiv preprint arXiv:2501.11858, 2025.
• Côté et al. [2019] Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pages 41–75. Springer, 2019.
• de Wynter [2024] Adrian de Wynter. Will gpt-4 run doom? IEEE Transactions on Games, pages 1–10, 2024.
• Deitke et al. [2022] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 35:5982–5994, 2022.
• Deng et al. [2021] Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to-end visual grounding with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1769–1779, 2021.
• Ding and Ito [2024] Shiyao Ding and Takayuki Ito. Minellama: Llama with retrieval-augmented generation as a decision maker in minecraft. In International Conference on Practical Applications of Agents and Multi-Agent Systems, pages 97–108. Springer, 2024.
• Ding et al. [2024] Xinpeng Ding, Jianhua Han, Hang Xu, Xiaodan Liang, Wei Zhang, and Xiaomeng Li. Holistic autonomous driving understanding by bird’s-eye-view injected multi-modal large models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13668–13677, 2024.
• Driess et al. [2023] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: an embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning. JMLR.org, 2023.
• Fan et al. [2024] Jiaqi Fan, Jianhua Wu, Jincheng Gao, Jianhao Yu, Yafei Wang, Hongqing Chu, and Bingzhao Gao. Mllm-sul: Multimodal large language model for semantic scene understanding and localization in traffic scenarios, 2024.
• Fei et al. [2024] Hao Fei, Xiangtai Li, Haotian Liu, Fuxiao Liu, Zhuosheng Zhang, Hanwang Zhang, and Shuicheng Yan. From multimodal llm to human-level ai: Modality, instruction, reasoning and beyond. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11289–11291, 2024.
• Guss et al. [2019] William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: a large-scale dataset of minecraft demonstrations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 2442–2448, 2019.
• Jose et al. [2024] Sagar Jose, Khanh TP Nguyen, Kamal Medjaher, Ryad Zemouri, Mélanie Lévesque, and Antoine Tahan. Advancing multimodal diagnostics: Integrating industrial textual data and domain knowledge with large language models. Expert Systems with Applications, 255:124603, 2024.
• Kolve et al. [2017] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017.
• Li et al. [2024a] Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends® in Computer Graphics and Vision, 16(1-2):1–214, 2024a.
• Li et al. [2024b] Hao Li, Xue Yang, Zhaokai Wang, Xizhou Zhu, Jie Zhou, Yu Qiao, Xiaogang Wang, Hongsheng Li, Lewei Lu, and Jifeng Dai. Auto mc-reward: Automated dense reward design with large language models for minecraft. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16426–16435, 2024b.
• Li et al. [2024c] Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks. In Advances in Neural Information Processing Systems, pages 49881–49913. Curran Associates, Inc., 2024c.
• Lin et al. [2021] Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang. Juewu-mc: Playing minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint arXiv:2112.04907, 2021.
• Liu et al. [2024] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. In European Conference on Computer Vision, pages 126–142. Springer, 2024.
• Lu et al. [2022] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022.
• Majumdar et al. [2024] Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16488–16498, 2024.
• Mangalam et al. [2023] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212–46244, 2023.
• Mokady et al. [2021] Ron Mokady, Amir Hertz, and Amit H. Bermano. ClipCap: CLIP prefix for image captioning. ArXiv preprint, abs/2111.09734, 2021.
• Ni et al. [2025] Fei Ni, Jianye Hao, Shiguang Wu, Longxin Kou, Yifu Yuan, Zibin Dong, Jinyi Liu, MingZhi Li, Yuzheng Zhuang, and YAN ZHENG. Peria: Perceive, reason, imagine, act via holistic language and vision planning for manipulation. Advances in Neural Information Processing Systems, 37:17541–17571, 2025.
• OpenAI [2025] OpenAI. Computer-using agent: Introducing a universal interface for ai to interact with the digital world. 2025.
• Qian et al. [2024] Cheng Qian, Peixuan Han, Qinyu Luo, Bingxiang He, Xiusi Chen, Yuji Zhang, Hongyi Du, Jiarui Yao, Xiaocheng Yang, Denghui Zhang, Yunzhu Li, and Heng Ji. Escapebench: Pushing language models to think outside the box, 2024.
• Qin et al. [2024] Yiran Qin, Enshen Zhou, Qichang Liu, Zhenfei Yin, Lu Sheng, Ruimao Zhang, Yu Qiao, and Jing Shao. Mp5: A multi-modal open-ended embodied system in minecraft via active perception. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16307–16316. IEEE, 2024.
• Savva et al. [2019] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
• Shridhar et al. [2020] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
• Wang et al. [2023a] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023a.
• Wang et al. [2025a] Junyang Wang, Haiyang Xu, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v: Learning mobile device operation through video-guided multi-agent collaboration, 2025a.
• Wang and Zhao [2023] Yuqing Wang and Yun Zhao. Gemini in reasoning: Unveiling commonsense in multimodal large language models. arXiv preprint arXiv:2312.17661, 2023.
• Wang et al. [2024] Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning. arXiv preprint arXiv:2401.06805, 2024.
• [37] Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, and Yitao Liang. Rat: Retrieval augmented thoughts elicit context-aware reasoning and verification in long-horizon generation. In NeurIPS 2024 Workshop on Open-World Agents.
• Wang et al. [2023b] Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560, 2023b.
• Wang et al. [2025b] Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(3):1894–1907, 2025b.
• Wang et al. [2025c] Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. Mobile-agent-e: Self-evolving mobile assistant for complex tasks. arXiv preprint arXiv:2501.11733, 2025c.
• Xie et al. [2025] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094, 2025.
• Xu et al. [2023] Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.04658, 2023.
• Yan et al. [2025] Yibo Yan, Shen Wang, Jiahao Huo, Jingheng Ye, Zhendong Chu, Xuming Hu, Philip S Yu, Carla Gomes, Bart Selman, and Qingsong Wen. Position: Multimodal large language models can significantly advance scientific reasoning. arXiv preprint arXiv:2502.02871, 2025.
• Yang et al. [2024a] John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024a.
• Yang et al. [2024b] Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14171, 2024b.
• Yang et al. [2025] Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502.09560, 2025.
• Yang et al. [2019] Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. A fast and accurate one-stage approach to visual grounding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4683–4693, 2019.
• Yang et al. [2023] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023.
• Yu et al. [2016] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016.
• Zhu et al. [2024] Yichen Zhu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Retrieval-augmented embodied agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17985–17995, 2024.

\thetitle

Supplementary Material

Appendix A Full Takeaway Observations

• •
  Distinct human-like behavioral patterns: There are distinct behavioral patterns among different models in the room escape task. GPT-4o and Gemini exhibit the most human-like behavior, but the patterns differ from each other. For example, GPT-4o demonstrating stable viewpoint control and deliberate movements. At beginning steps, it tends to walk around with large distances and observe the surroundings in a wide range. While Gemini tends to look around before taking further actions to move to other locations. Some other models tend to shift their perspectives arbitrarily, leading to inefficient exploration. Gemini frequently gets stuck, struggling with spatial navigation, while Phi-3 exhibits a tendency to continuously rotate in place with minimal actual movement, hindering effective exploration.
• •
  Robustness towards system prompts: Most of the models are faithful to the system instruction. They are aware of the ultimate target, to locate the door and make a way out, and are not easily distracted by findings during the exploration, except for Phi-3 who always fail to generate required actions with valid and interactable items.
• •
  Common failure modes: However, significant limitations persist across all model, including GPT-4o sometimes. A common failure mode is inaccurate object positioning within the field of view. Models often fail to center the target object, which is indicated by a guiding red dot in our environment, precisely suggesting the objects to interact with. This often leads to unsuccessful interactions such as grabbing or entering for password. Furthermore, some models struggle with tool utilization, particularly in cases requiring abstract reasoning, such as correctly applying a password to unlock a door.

Appendix B Human Evaluation

Table 6: Detailed results for various levels.Since humans completed all escape tasks in the evaluation, the escape rate is 100% and not reflected above.

We conduct manual evaluation on MM-Escape, and report detailed results in Table 6. Human participants in EscapeCraft exhibit a clear understanding of how to complete tasks efficiently. By observing objects in the environment, they can make reasonable judgments about which items to pick up, leading to a higher success rate in effective item acquisition and usage. Additionally, when unable to open doors or interactable objects, humans are more adept at promptly shifting their approach to seek alternative clues in the environment rather than getting stuck. In terms of spatial awareness, they demonstrate a strong ability to perceive the relative positions of objects, enabling logically reasonable, smoother and more coordinated exploratory actions. Across Difficulty-1 to 3, human participants consistently identify all necessary items with less interaction attempts compared with MLLMs, and successfully complete all the tasks within a limited number of steps.

Appendix C Construction Details

C.1 Environment Construction

C.1.1 Room Generation

We adopted the automated 3D room generation method ProcTHOR [8], with additional improvements to enhance its flexibility and applicability regarding diverse type of scenes. Following Procthor, we generate 3D environments that can simulate diverse real-world scenes, such as bedrooms, living rooms, and offices by maintaining collections of typical objects that are common in different scenes. For instance, desks in offices, workbenches in laboratories, and other representative objects of corresponding scenes. We enable automatic creation of 3D rooms from the collections of each scenes, ensuring that the generated rooms accurately reflect their respective environments.

We use a configuration file to generate each room, specifying the items along with required styles, positions, sizes, and interactivity. This enables precise control over the placement of prop objects, ensuring that they are arranged in a manner aligning with real-world expectations on spatial arrangement.

Benefits of the Automated 3D Room Generation include:

• •
  Diversity and Complexity: By automatically generating a variety of 3D rooms, we can provide the model with diverse environments, ensuring that it is capable of handling various layouts, objects, and puzzle elements. This diversity is critical in assessing the model’s ability to reason in different scenarios, evaluating its performance when confronted with unknown and complex situations. Moreover, the ability to create different configurations on the fly means the model will not be limited to predefined environments, which helps to prevent overfitting to specific room layouts.
• •
  Enhanced Realism: Unlike manually designed fixed scenes, automatically generated 3D environments can simulate more natural and irregular spatial layouts. This is essential for training and evaluating agents on spatial reasoning, pathfinding, and interaction skills. By incorporating a wide range of room designs, we create more realistic scenarios in which agents must navigate, interact with objects, and solve problems, similar to real-world challenges.
• •
  Efficiency: The automated generation of 3D rooms significantly improves the efficiency of the testing process. Whether for debugging model performance or conducting large-scale evaluations, the ability to generate various environments quickly eliminates the time-consuming and tedious process of manually creating scenes. This allows for faster iteration and more comprehensive testing without the bottleneck of scene creation.
• •
  Evaluation Robustness: In escape room-style games, the diversity of room layouts and puzzles directly influences the game’s difficulty. By automating the scene generation process, we enable the model to train and be evaluated in a wide array of environments, which helps enhance its robustness. This diversity allows the model to develop better strategies for handling new and unexpected challenges, improving its ability to generalize across different scenarios.
• •
  Prevention of Cheating and Overfitting: One of the major advantages of generating an infinite variety of scenes is the prevention of both cheating and overfitting. Fixed testing environments often lead to overfitting, where a model can ”learn” to exploit certain patterns or repetitive features of the environment. In contrast, each automatically generated room is unique, with random elements that require the model to demonstrate true problem-solving abilities in previously unseen configurations. This ensures that the model cannot simply memorize the environment but must adapt its strategies to succeed.
• •
  Adaptive Adjustment: Another key feature of our approach is the ability to dynamically adjust the agent’s starting position and other environmental variables. This feature allows us to test how the agent performs under different initial conditions, such as varying the agent’s starting location, the distribution of objects, or the complexity of the puzzle. These adjustments enable a more comprehensive assessment of the agent’s performance, providing deeper insights into its ability to adapt and solve problems in diverse situations.

The automated 3D room generation framework we developed not only enhances the diversity and realism of testing environments but also optimizes the efficiency of large-scale evaluations. By providing a mechanism for dynamically altering the environment and agent conditions, it offers a more robust and fair evaluation process, ensuring that models are evaluated under realistic, varied, and challenging conditions.

C.1.2 Action Space

In our EscapeCraft environment, the agent is allowed to perform a set of actions that facilitate its interaction with the environment. These actions include moving forward, rotating right, rotating down, looking at specific coordinates, grabbing objects, and interacting with elements in the environment. Each of these actions plays a distinct role in enabling the agent to explore and solve tasks within the escape scenario.

• •
  Moving Forward: This action specifies the distance the agent needs to travel along its current heading. The agent’s movement is controlled by the distance parameter, which dictates how far it should move in a straight line.
• •
  Rotating Right: This action specifies the angle by which the agent should rotate to the right. The agent can adjust its orientation by a specified angular increment, which allows it to navigate through the environment by changing its field of view.
• •
  Rotating Down: Similar to rotating right, this action allows the agent to lower its head by a specified angle. This action is crucial for examining objects at different vertical levels, contributing to a more thorough exploration of the environment.
• •
  Looking At: The ”looking at” action involves orienting the agent’s view towards a specific coordinate within its current field of view. This coordinate is represented in a relative manner, with the center of the field of view denoted as (0.5, 0.5). By specifying the target coordinates, the agent can focus on particular objects or areas of interest in the environment.
• •
  Grabbing: The grabbing action indicates if the agent wants to pick up an object or interact with an item within its proximity. This action is typically used when the agent identifies an object that can be picked up or manipulated, allowing it to add that item to its inventory or interact with it to get crucial information.
• •
  Interacting: The interaction action is multifaceted and depends on the context of the object the agent is engaging with. Interactions fall into three primary categories:
  1. 1. Item Usage: The agent can use items from its inventory by referencing the unique ID of an item, such as a key, tool, or piece of equipment that it has previously obtained. In these cases, the agent specifies the item ID and applies it to relevant objects in the environment (e.g., using a key to unlock a door).
  2. 1. Text Input: Some interactions require the agent to input text, such as a password to unlock a combination lock. These textual inputs are necessary to progress in the environment when dealing with specific security mechanisms.
  3. 1. Read: When the agent wants to know the detailed information of an item in its inventory (e.g., reading the content recorded in a note), it can use this parameter and provide the ID of the corresponding item to the item usage field.
        A special case arises when the agent performs the grab action and leaves the interaction input empty. In this instance, it indicates the agent’s intent to pick up an item within the field of view, without specifying a particular item to interact with. This action is used when the agent is trying to collect objects that are relevant to its escape mission.

Throughout the agent’s exploration, its interactions with the environment yield varying types of feedback. The environment is populated with different types of objects, classified as follows:

• •
  Non-Escape Related Props: These are objects within the environment that do not directly contribute to the agent’s escape objectives. Interactions with these items provide no information or progress.
• •
  Collectible Items: These items can be obtained and added to the agent’s inventory, providing critical information and/or utility for the agent’s tasks. Upon collection, the agent gains knowledge of the item’s identity and its associated attributes.
• •
  Locked Props: These include objects such as locked doors, chests, or other secured items. When the agent interacts with a locked object in the early stages, it receives a prompt indicating the type of item required to unlock it. Upon obtaining the corresponding item (e.g., a key, a code, or another unlocking mechanism), the agent can use the appropriate item from its inventory to unlock the object by specifying its ID or providing the required input (e.g., entering a password). Once these items are unlocked, the agent will immediately obtain the props contained in them and be informed of the simple information of the items obtained.

These interaction dynamics are crucial for the agent’s progression in the environment, as they form the basis for decision-making, object management, and problem-solving. The design of these interactions reflects the need for both exploration and strategy, with the agent needing to acquire, manage, and apply various items in order to navigate and ultimately escape the environment.

C.2 Data Construction

C.2.1 Prop Chain

We proposed a procedural generation approach for constructing game settings tailored to overcome the inherent limitations of current language models, such as restricted context length and reduced reasoning capabilities. To address these constraints, we propose the concept of Prop Chain, a singly linked list that organizes interactive game elements in a sequence, ensuring a coherent flow of gameplay interactions. Each node in the linked list corresponds to a distinct interactive item or action, such as a key, a locked box, or a note with a password. The tail node of the chain signifies the game’s exit point, thereby serving as the conclusion of the sequence. Table 7 shows the the Prop Chain for the Difficulty-3 Level.

In our implementation of the Prop Chain, we initially focus on a set of fundamental game elements: a key, a locked box (which can only be opened with a key or password), a note (carrying both password and story-related information), and an exit (which is locked and requires either a key or password to access). These components are used to construct a series of interconnected nodes, where each item or action is represented by a node in the chain. The links between the nodes define the relationships between the props and the ways in which they can be obtained or used during the game. For instance, some props may be freely accessible, while others require specific conditions, such as possessing a key to unlock a box, or using a specific password to open the door.

The inter-node relationships can be annotated to represent different interaction modes. For example, a key can be placed within a box, requiring the player to first unlock the box before acquiring the key. Additionally, nodes are allowed to contain multiple conditions. A note revealing the password to the exit are both narrative (to see or infer the textual password) and functional (to open the door).

Each node has an additional show property set to indicate whether the item should appear directly in the scene (for example, a key placed in a box only needs to show the box in the scene, while a key that can be directly obtained independently needs to be shown in the scene), allowing us to determine which props need to be generated in the 3D scene by reading the game settings.

While our initial focus on a limited set of props and interactions, such as the key, locked box, note, and exit, suffices for creating a variety of escape game settings that challenge current language models, the system is highly extensible. The procedural nature of Prop Chain allows for the seamless integration of new props, interactions, and unlocking mechanisms. As such, the framework can easily accommodate additional types of interactive items, more intricate unlock conditions, and customized gameplay mechanics in future iterations. This scalability ensures that the approach remains adaptable to more complex and diverse game scenarios, further enhancing its applicability for testing language models in a variety of settings.

The Prop Chain framework provides a robust and flexible methodology for the procedural generation of game settings. By focusing on a set of core interactive elements and defining their relationships within a linked list structure, we have developed a scalable approach that can evolve to incorporate new game dynamics and meet the increasing demands of future language models.

Table 7: Representation of the Prop Chain for the Difficulty-3 Level. The level includes a sequence of interactive props where only box_1 and note_1 are visible in the room. The gameplay progression follows a structured sequence: the agent first discovers note_1, which contains the password_1 needed to unlock box_1. Inside box_1, the agent retrieves key_1 and note_2, the latter of which contains a story element of the game. Finally, the agent uses key_1 to unlock the exit and complete the game.

Appendix D Analysis of Moving Distance

Table 8: Correlation between optimal distance and model moving distance.

We calculate the optimal distance required for escape tasks in each scene and compare it with the real distance experienced by the models. Contrary to our expectations, the experienced distance does not exhibit a significant correlation to the distance among key props and the exit within the scene shown in Table 8. This discrepancy may be attributed to the lack of holistic environmental perception of models, which prevents them from further reasoning and planning based on current and ultimate goals, thereby failing to generate an effective and optimal route to complete the task.

Appendix E Analysis of Grabbing Behaviors

In Figure 4 (b)(c)(d), we analyzed three performance metrics, steps, GRS, and Rgrabsubscript𝑅grabR_{\text{grab}}italic_R start_POSTSUBSCRIPT grab end_POSTSUBSCRIPT, during the model task completion process under Visibility of Exits at initial locations and orientations . The results indicate that, under common trends, the ability to see the exit from the initial position aids the model in escaping the room with fewer steps.It aligns with our intuition, as the exit, crucially related to the ultimate task goal, plays a significant role in model’s visual recognition, reasoning and interaction with the environment to collect information. However, there exist exceptions. For Difficulty-1, some well-performed models still struggle to achieve high GSR and consume more steps despite being able to see the exit from their initial location. They do not interact directly with the exit at the very beginning, and instead choose to rotate around explore the environment for more information and to conduct reasoning and taking action. This is also evident in Difficulty-2 and -3, where these models, after acquiring the key prop, can locate the exit and escape more efficiently, as reflected in better performance in terms of GRS, Rg⁢r⁢a⁢bsubscript𝑅𝑔𝑟𝑎𝑏R_{grab}italic_R start_POSTSUBSCRIPT italic_g italic_r italic_a italic_b end_POSTSUBSCRIPT and steps.

We further raise three questions for the analysis of the reasoning process during escaping: i) How many steps it costs to obtain props? ii) How many steps it costs to exit the room after obtaining the core prop (key or password to the door)? iii) What is the relationship between grab success rate (GSR) and escape outcome for each test?

For question 1, GPT-4o demonstrates a significant advantage in the number of steps required to obtain the key followed by Gemini as shown in Table 4. Although Claude requires fewer average steps to find props in Difficulty-3, this comes at the cost of a significant decrease in escape rate. The superior performance in locating and obtaining the core prop can be attributed to model’s better understanding of task objectives and the environment in the escape room, as well as its enhanced reasoning abilities in this context.

For question 2, Gemini is able to locate and acquire the key at lower cost in difficulty-2. But in difficulty-3 which is more complex, GPT-4o performs better. It finds the core prop with fewer steps and its prior memory and understanding of the room environment—gained in the process of obtaining key props—aids it to locate the exit and escape using even fewer steps compared to other models.

For question 3, we observe that escape success is positively correlated with GSR, as shown in Figure 4(a). A higher Grab SR implies that models have experienced more successful interactions with the environment. It potentially indicates a clearer understanding of the overall environment and ultimate goals within the room escape task, leading to a higher success rate. As difficulty increases, the Grab SR of most models declines, and many of them fail to escape. However, GPT-4o and Claude 3.5 remain relatively stable, with less variation in grabbing behavior and success rate across difficulty settings compared to others. The low success rate of Qwen, and Llama 3.2 11B in difficulty 2 and 3 can be partly attributed to their inability to effectively perceive the environment, reason and make appropriate object interaction choices in more complex tasks.

Table 9: Detailed results of note-key and key-note settings of Difficulty-3.

Appendix F Discussion of Fully Autonomous Multi-room Escape

We discussed a simplified multi-room setting in Table 3. We further study how models behavior in this section. The ER of GPT-4o decreases to only 50% on average for the settings of applying Difficulty-2 to room 2. The grabbing behaviors also change, where both the Grab SR and Grab Ratio decreases. Similar trends are observed for Gemini and Claude. These indicate that models can learn from a successful escape history. We also note that by setting the two rooms to the same difficulty level further helps models to escape, while different levels do not benefit as expected.

Appendix G Discussion of Customizing Difficulties

We enable two different settings of Difficulty-3, a key-note setting and a note-key setting. We observe that human annotators perform equally for both settings (from Table 6), while some models present preferences regarding the key-first and the note-first (i.e. the password-first), as shown in Table 9. Gemini presents an approaching GPT-4o level results in the note-key setting, while scores the worst in all calculated metrics among the three reported models, presenting a preference towards searching for the note rather than recognizing and interacting with the key. Additionally, Claude scores higher in Grab SR regarding the key-first setting than the note-first setting, potentially indicating a better attention on the key (directly used to unlock the door) than on the note (with clues, implicitly assists with the escape process).

For the multi-room setting, whose results are reported in Table 3, we further extend the experiments to a full autonomous scenario to require models to escape both room in order all by themselves. This means, the first room no longer serves as a bootstrapping guidance. We notice a performance drop both in the escape rate and the grabbing behaviors.

Appendix H Experiments with Reasoning Models

Results of recently released reasoning models, such as Claude 3.7 and o1, are reported in Table 10. We also provide results fo GPT-4o and Claude 3.5 for comparison. Notably, o1 and Claude 3.7 attempt fewer grabs but yield higher GSR and Prop Gain, indicating more efficient and intelligent reasoning compared to there previous versions.

Table 10: Results of reasoning models on Difficulty-2.

Appendix I Results of Post-game Debriefing

We choose models with top-2 escaping rate for evaluation, that is, GPT-4o and Gemini-1.5-pro. We first leverage the debriefing prompt, as shown in Table 16, to get recovered stories from models. Then we use GPT-4o to assess the recovered stories according to their similarities to our given groundtruth stories, then assign a score between 0 and 5 to each recovered story. As shown in Table 11, average scores of both models are low. We further show a case in Table 12. We can see that models can recall and describe their escape process relatively well, but they do not remember clues about the background stories well, which leads to the failure of restoring the full picture of stories.

Table 11: Average scores of models on post-game debriefing task. Scores are between 0 and 5.

Table 12: A case of post-game debriefing.

Appendix J Required Abilities

In our MM-Escape Benchmark, the free-form task challenges models with the following abilities:

• •
  Visual search. The model should have a prior knowledge on the potential locations of useful clues and objects, reason about its current field of view and make decisions to explore the environment efficiently.
• •
  Active perception. Due to the limited field of view, the model cannot acquire all relevant information at once. It must actively adjust the perspective by moving and rotating its viewpoint, allowing is to perceive from different angles and effectively identify key objects, such as the key, the password and the door.
• •
  Spatial reasoning. The model is required to deduce spatial depth and objects in the room. It must infer hidden or obscured elements, estimate distances, and determine the most appropriate actions based on the spatial configuration of its surroundings it observed. The model should leverage its understanding of space to make decisions about movement, interaction, and problem-solving, progressively advancing toward the escape objective.
• •
  Target deduction. The model is only given the objective to escape the room, without explicit instructions on how to achieve it. Hence, it should autonomously infer necessary actions through interaction and exploration, or decompose the ultimate objective into feasible and short-term goals.
• •
  Prop Utilization Capability. In the escape room environment, the model must maintain awareness of the items in its inventory and determine the appropriate moments to use them. Effective utilization of these props is crucial for navigating the complex environment and achieving the escape objective.
• •
  Long-term reasoning. Successfully escaping the room requires a prolonged sequence of interactions. The model must analyze and integrate long-form text-image data across multiple key interaction steps to make informed decisions.

Appendix K Prompt Template

System Prompt

The System Prompt consists of two primary components: the Instruction Prompt and the Operation Prompt. The Instruction Prompt provides the model with contextual information regarding the current environment, its overarching objective, and the approach required to achieve this objective. In contrast, the Operation Prompt delineates, in precise detail, the permissible actions and exploratory methods that the model can employ within the environment. Additionally, it specifies the format and structure of the structured data that the model is expected to generate in response. The complete prompt is shown in table 13.

Table 13: The System Prompt

Step Prompt

The Step Prompt is designed to provide feedback to the model regarding the outcome of its previous interaction with the environment (if an interaction was attempted). Simultaneously, it informs the model in real-time about the items currently available in its inventory for potential use. Additionally, the prompt serves as a directive, encouraging the model to continue exploration or engage in further interactions.The complete prompt is shown in table 14.

Table 14: The Step Prompt

Prompt for Consistency Evaluation

The Consistency Evaluation Prompt is designed to assess whether the multimodal agent’s reasoning aligns with the actual outcomes of its actions during an escape room interaction. After each interaction, the model is given the agent’s internal rationale, describing its belief or intended action, and the environment’s response, which records what actually occurred. The prompt guides the model to judge if the target object mentioned in the rationale matches the object that was truly interacted with, thereby evaluating whether the behavior is intentional or accidental. A special case is defined for successful escapes: the rationale must explicitly or implicitly indicate the agent’s goal to exit the room. The model outputs a binary judgment in JSON format, indicating consistency (‘1’) or inconsistency (‘0’). The complete prompt is shown in table 15.

Table 15: The Consistency Evaluation Prompt

Debriefing Prompt

The Story Recovery Prompt is used to guide the model to recall and infer the background and story of the entire game based on the interaction records after the model successfully escapes the room. The model is guided to describe the room environment, recall the items that may contain information or clues, and finally piece together the whole story to complete the story recovery. The complete prompt is shown in table 16.

Table 16: The Story Recovery Prompt