Kaijie Yang - Openstream.ai | LinkedIn (original) (raw)

#story How confessions can keep language models honest Open AI shares an early proof-of-concept method that trains models to report when they break instructions or take unintended shortcuts. https://lnkd.in/gy9TnHsV https://lnkd.in/g9gUnHNz This “confessions” method surfaces hidden failures—guessing, shortcuts, rule-breaking—even when the final answer looks correct. A confession is a second block of text that comes after a model’s main response to a request, in which the model marks itself on how well it stuck to its instructions. The idea is to spot when an LLM has done something it shouldn’t have and diagnose what went wrong, rather than prevent that behavior in the first place. Studying how models work now will help researchers avoid bad behavior in future versions of the technology, One reason LLMs go off the rails is that they have to juggle multiple goals at the same time. Models are trained to be useful chatbots via a technique called reinforcement learning from human feedback, which rewards them for performing well (according to human testers) across a number of criteria. “When you ask a model to do something, it has to balance a number of different objectives—you know, be helpful, harmless, and honest,” says Barak. “But those objectives can be in tension, and sometimes you have weird interactions between them.” To train an LLM to produce confessions, Barak and his colleagues rewarded the model only for honesty, without pushing it to be helpful or helpful. Importantly, models were not penalized for confessing bad behavior. “Imagine you could call a tip line and incriminate yourself and get the reward money, but you don’t get any of the jail time,” says Barak. “You get a reward for doing the crime, and then you get an extra reward for telling on yourself.” The OpenAI team is up-front about the limitations of the approach. Confessions will push a model to come clean about deliberate workarounds or shortcuts it has taken. But if LLMs do not know that they have done something wrong, they cannot confess to it. And they don’t always know. #AI #guessing #shortcuts #RuleBreaking #ReinforcementLearning