PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking Thanks: Citation: M.J. Buehler, et al., PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking. Pages…. DOI:000000/11111.

摘要

We introduce PRefLexOR (Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning), a framework that combines preference optimization with concepts from Reinforcement Learning (RL) to enable models to self-teach through iterative reasoning improvements, to create synthetic intelligence with enhanced scientific reasoning capabilities. Central to PRefLexOR is a recursive approach that engages the model in multi-step reasoning, revisiting, and refining intermediate steps before producing a final output in both training and inference phases. The foundation of PRefLexOR lies in multi-stage training, where the model first learns to align its reasoning with scientifically accurate decision paths by optimizing the log odds between preferred and non-preferred responses through a novel in-situ dataset generation algorithm. For on-the-fly training data generation, PRefLexOR builds a dynamic knowledge graph by generating questions from random text chunks and utilizing retrieval-augmentation to contextualize relevant details from across the entire corpus, resulting in rigorous reasoning chains. In a second stage, preference optimization strategies further enhances model performance by using rejection sampling to fine-tune reasoning quality by continually producing in-situ training data while masking the reasoning steps to focus on discovery of novel mechanisms to achieve correct answers. This hybrid approach mirrors key aspects of RL, where the model is continuously guided by feedback to improve decision-making and reasoning, and the adaptive process enables the model to self-teach as it continually improves through real-time feedback and recursive processing. Our method does not use pre-generated datasets and instead trains the model to continuously adapt and improve in real time. Recursive optimization within special thinking tokenization introduces iterative feedback loops, where the model refines its reasoning, much like policy refinement in RL, achieving deeper coherence, consistency, and adaptability. By recursively optimizing reasoning through feedback-driven learning, PRefLexOR achieves significant flexibility in its ability to handle complex tasks, learning and evolving its cognitive abilities autonomously. PRefLexOR’s recursive optimization mirrors how biological systems adapt and evolve. By using feedback loops to refine reasoning pathways during training and/or inference, it emulates nature’s resilience and adaptability, enhancing its decision-making capabilities. Implemented in very small language models with only 3 billion parameters, we showed that even tiny models can iteratively teach themselves to reason with greater depth and reflectivity, akin to an RL-based self-improving system capable of solving open-domain problems with superior reasoning depth and logic. Our implementation is straightforward and can be incorporated into any existing pretrained LLM. We focus our examples on applications in biological materials science, and demonstrate the method in a variety of case studies that range form in-domain to cross-domain applications. We explore several reasoning strategies that include both thinking and reflection modalities to construct a multi-agent recursive self-improving model that can successively improve responses via repeated sampling during inference, offering flexibility and integration into larger agentic systems.