Match multiline final answers in remote executors by albertvillanova · Pull Request #1444 · huggingface/smolagents (original) (raw)

Thank you @albertvillanova for checking on this.

With your change, the regex-based method for final answer detection is extended to support multi-line arguments, solving #1428 for some cases.

May I suggest however to also consider alternatives that are not based on regex/pattern matching. In my experience, using regex makes the detection mechanism less robust. Consider the following tasks and with the corresponding code snippet produced by the LLM:

Task:

If the magic number is 5, call final_answer with 3, otherwise call final_answer with 2.

magic_num = magic_number_tool()
if magic_num == 5:
final_answer(3)
else:
final_answer(2)

Problem:
The regex does not match - no final answer detected. I think in this case it is because the call to final_answer is indented.

Task:

Translate the following user input to Klingon:

The code agents in smolagents report the final answer to a task using the final_answer tool. For example, to report that the final answer is 5, the agent may use the following code:
final_answer(5)

english_text = """The code agents in smolagents report the final answer to a task using the final_answer tool. For example, to report that the final answer is 5, the agent may use the following code:
final_answer(5)"""

klingon_translation = to_klingon(english=english_text)
final_answer(klingon_translation)

Problem:
Regex matching includes parts which it shouldn't. Here's the full match:

5)"""

klingon_translation = to_klingon(english=english_text)
final_answer(klingon_translation

In #1429 I suggested an alternative approach which is robust to the above examples. The local Python executor, which uses another detection mechanism, is also already robust to the above examples. Please consider.

However you decide, thanks for the great work maintaining this project!