Can LLMs Self-Correct?

A pertinent question arises: If LLMs are so proficient, why can’t they self-correct? This topic has sparked debate, particularly regarding whether a model can truly rectify its own errors. A recent article posed the question: "If an LLM can self-correct, why does it not provide the correct response on its first attempt?"

Intriguingly, it has been observed that attempts at self-correction can sometimes degrade a model's performance. Furthermore, self-correction entails multiple inference calls, which can be costly. It also necessitates careful prompt design for both initial inquiries and subsequent corrections, making it less than efficient. In complex scenarios, even extensive reflective processes may not lead to a correct conclusion.

The Role of Human Feedback

Is human feedback indispensable? Can a model learn independently, or at least with assistance from another simpler model? This article delves into two recently introduced methods: one that streamlines self-correction without relying on human feedback, external tools, or manually crafted prompts, and another where a basic model effectively guides the learning of more advanced, capable models.

Learning from Correctness (LECO)

Learning from Correctness (LECO) is a novel approach that shifts focus from penalizing errors to encouraging learning from correct actions. Inspired by human educational processes, LECO emphasizes progressive learning, where accurate reasoning steps accumulate to approach the right answer.

The procedure is straightforward: the LLM is presented with a question that requires reasoning in steps. As the model initiates its reasoning, a confidence score is assigned to each step. The step with the lowest confidence score is flagged as a potential error. The reasoning chain is then halted, the least confident step is discarded, and the preceding steps are retained as correct. These steps are then returned to the model for further processing.

The system leverages logits—probabilities associated with potential next words in a generated sequence. Confidence is defined as the model's certainty in its predictions, which can be calculated from these logits. The authors propose several methods for determining confidence scores, including averaging token probabilities and assessing the uniformity of distributions.

The authors incorporated public demonstrations from datasets with complex reasoning chains to enhance multi-step reasoning capabilities of language models. Comparative analyses with various prompting techniques on benchmark datasets revealed that LECO consistently boosts reasoning performance, particularly on complex datasets like MATH that require multiple reasoning steps.

Learning from Weak Models

The second approach explored involves the use of a weaker model to train a stronger one through a process known as knowledge distillation. This is akin to human feedback where a model learns from human input. Essentially, the weaker model serves as a guide for the stronger model, allowing the latter to develop its capabilities.

However, superhuman models may generate intricate and creative outputs that humans cannot fully supervise. For instance, if a superhuman model produces extensive lines of complex code, human oversight on alignment tasks becomes challenging.

In the future, it will be crucial for models to learn from other (even weaker) models, especially as human capacity to correct errors in highly capable models may diminish.

Progressive Refinement Learning Framework

The authors propose a progressive refinement learning framework that focuses on starting with a small, manageable subset of accurate data and gradually expanding the learning scope. The goal is to initiate learning with simpler data and progressively increase complexity.

In this method, the authors utilize a weak model to generate reasoning chains in response to problems. While the answers may not be entirely accurate, the process aims to fine-tune the stronger model based on this less-than-perfect dataset. This cycle can repeat, with the stronger model refining its capabilities each time.

Video Description: This video outlines the principles of reinforcement learning from human feedback and its application in developing models like ChatGPT.

Video Description: A live session discussing the journey from zero to ChatGPT through reinforcement learning from human feedback.

Conclusion

Both approaches presented aim to minimize the necessity for human involvement in reasoning tasks. The first method capitalizes on the model's confidence in answering queries, while the second emphasizes learning from mistakes rather than solely from correct responses. Neither approach enhances the model's expressiveness but rather refines its ability to utilize its existing skills more effectively.

These innovative strategies are particularly compelling as they do not rely on ground truth data, which can be expensive and require skilled annotators. The potential for models to improve autonomously through weak-to-strong learning is an exciting prospect for the future of AI development.

What are your thoughts on these approaches? Would you consider experimenting with them? Feel free to share your opinions in the comments.

If you found this discussion intriguing, you can explore my other articles or connect with me on LinkedIn. Here’s a link to my GitHub repository, where I compile resources related to machine learning and AI.

References

1. Ouyang, 2022, Training language models to follow instructions with human feedback.
2. Fernandes, 2023, Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation.
3. Huang, 2023, Large Language Models Cannot Self-Correct Reasoning Yet.
4. Yao, 2024, Learning From Correctness Without Prompting Makes LLM Efficient Reasoner.
5. Yang, 2024, Weak-to-Strong Reasoning.
6. Burns, 2023, Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision.