Announcing the RLC 2025 Outstanding Paper Awards

We are honoured to announce the winners of the Outstanding Paper Awards at the Second Reinforcement Learning Conference. Like last year, papers are awarded based on specific aspects of their contribution. We describe the process we used to select the awarded papers below.

This year's awards consist of eight papers. The awarded papers are listed below in alphabetical order by award name. Congratulations to all the authors!

Applications of Reinforcement Learning

WOFOSTGym: A Crop Simulator for Learning Annual and Perennial Crop Management Strategies

Authors: William Solow, Sandhya Saisubramanian, Alan Fern

This is the first paper to introduce a high-fidelity crop simulation environment that uniquely supports both annual and perennial crops in multi-farm settings. This work addresses a critical gap in RL applications for agriculture, enabling the development of advanced agromanagement strategies under realistic constraints. Its innovative modifications to the WOFOST model and use of Bayesian Optimization significantly advance the practical utility and scientific rigor of RL in this vital domain.

Emerging Topics in Reinforcement Learning

Towards Improving Reward Design in RL: A Reward Alignment Metric for RL Practitioners

Authors: Calarina Muslimani, Kerrick Johnstonbaugh, Suyog Chandramouli, Serena Booth, W. Bradley Knox, Matthew E. Taylor

This paper makes an exceptional contribution to the emerging field of reinforcement learning from human feedback by introducing the Trajectory Alignment Coefficient, a new metric for evaluating how well a reward function aligns with human preferences. This work stands out for its practical impact, demonstrating that the metric reduces cognitive workload and increases the success rate of selecting performant reward functions for RL practitioners. By addressing the critical challenge of reward design in real-world applications, this paper offers a pioneering tool that is poised to significantly influence future research in the space of human-AI interaction.

Empirical Reinforcement Learning Research

Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functions

Authors: Ayush Jain, Norio Kosaka, Xinhu Li, Kyung-Min Kim, Erdem Biyik, Joseph J Lim

This paper addresses the fundamental challenge of local optima in complex Q-functions, a key problem for off-policy actor-critic methods in real-world applications. It proposes Successive Actors for Value Optimization (SAVO), an architecture that uses multiple actors and progressively simplified Q-landscapes to escape suboptimal policies. The work is a model of empirical rigor, demonstrating SAVO's superior performance across a diverse suite of challenging tasks, including dexterous manipulation and large-scale recommender systems.

Resourcefulness in Reinforcement Learning

PufferLib 2.0: Reinforcement Learning at 1M steps/s

Authors: Joseph Suarez

This paper introduces PufferLib 2.0, a resourceful toolkit that tackles the high computational cost of modern reinforcement learning research. By offering a suite of C-based environments and fast vectorization methods, PufferLib enables high-speed simulation, making large-scale experimentation accessible on a single desktop. This project's focus on a smooth and efficient development experience for complex environments marks a significant contribution to fostering frugal yet rigorous empirical research.

Scientific Understanding in Reinforcement Learning

Multi-Task Reinforcement Learning Enables Parameter Scaling

Authors: Reginald McLean, Evangelos Chatzaroulas, J K Terry, Isaac Woungang, Nariman Farsad, Pablo Samuel Castro

This paper significantly advances the scientific understanding of multi-task reinforcement learning by demonstrating that performance gains often attributed to complex architectures are primarily a result of parameter scaling. It reveals that naïvely scaling up a simple feed-forward architecture can outperform more sophisticated designs. Furthermore, the work uncovers a novel relationship where increasing task diversity can mitigate plasticity loss in larger models, providing a clear, evidence-based understanding of the drivers of performance in multi-task RL.

How Should We Meta-Learn Reinforcement Learning Algorithms?

Authors: Alexander David Goldie, Zilin Wang, Jakob Nicolaus Foerster, Shimon Whiteson

This paper provides a crucial empirical study that significantly advances the scientific understanding of how to meta-learn reinforcement learning (RL) algorithms. It directly compares several meta-learning approaches—including black-box learning, distillation, and LLM proposals—and offers a clear analysis of their trade-offs in terms of performance, sample cost, and interpretability. The findings provide actionable recommendations to help researchers design more efficient and effective approaches for meta meta-learning RL algorithms.

Theory of Reinforcement Learning

Deep Reinforcement Learning with Gradient Eligibility Traces

Authors: Esraa Elelimy, Brett Daley, Andrew Patterson, Marlos C. Machado, Adam White, Martha White

This paper presents a foundational theoretical contribution to deep reinforcement learning by extending the Generalized Projected Bellman Error (GPBE) to a multi-step objective, GPBE(λ), using λ-returns. It derives and evaluates three novel Gradient TD algorithms for this new objective, providing both forward-view and backward-view formulations compatible with modern deep RL practices. This work offers a principled approach to stable off-policy learning, tackling the divergence issues of semi-gradient methods and demonstrating superior performance in practice.

Tooling, Environments, and Evaluation for Reinforcement Learning

Syllabus: Portable Curricula for Reinforcement Learning Agents

Authors: Ryan Sullivan, Ryan Pégoud, Ameen Ur Rehman, Xinchen Yang, Junyun Huang, Aayush Verma, Nistha Mitra, John P Dickerson

This paper introduces Syllabus, a groundbreaking library that provides portable curriculum learning algorithms and infrastructure, addressing a critical gap in standard RL tooling. By defining a universal API, Syllabus enables researchers to easily integrate advanced curriculum learning methods into nearly any RL library. This work significantly lowers the barrier to entry for a crucial component of modern RL, encouraging research on more complex, challenging environments like NetHack and Neural MMO.

Award Categories

Following the first edition of RLC, the "Outstanding Paper Award" at RLC 2025 is different from what is traditionally done in machine learning conferences. We do not award papers for being the overall "best" papers in a conference, instead we award papers for making significant contributions to specific aspects of research. We believe such an approach will be more inclusive, it will celebrate the diverse types of scientific contribution one can make in the field, and it will give a more equal opportunity for different types of papers to be awarded. The idea is to award papers for excelling in what they propose to do.

This year, we considered awards for nine categories. Relative to last year, we introduced two new categories, one for RL contributions to natural sciences and one for contributions to emerging topics in RL.

We have also changed the name of one of last year's award categories from Outstanding Paper Award on Support Tools for RL Research to Outstanding Paper Award on Tooling, Environments, and Evaluation for Reinforcement Learning to make it more descriptive.

Finally, we refined some of the descriptions of last year's categories to make them more comprehensive and better reflect the breadth of contributions we aim to consider for that category. A more detailed description of each category is available at the end of this document. Importantly, no award is more prestigious than the other.

Full list of outstanding awards we considered for RLC 2025 (in alphabetical order):

Outstanding Paper Award on Applications of Reinforcement Learning
Outstanding Paper Award on Emerging Topics in Reinforcement Learning
Outstanding Paper Award on Empirical Reinforcement Learning Research
Outstanding Paper Award on Pioneering Vision in Reinforcement Learning
Outstanding Paper Award on Reinforcement Learning Contributions to Natural Sciences
Outstanding Paper Award on Resourcefulness in Reinforcement Learning
Outstanding Paper Award on Scientific Understanding in Reinforcement Learning
Outstanding Paper Award on the Theory of Reinforcement Learning
Outstanding Paper Award on Tooling, Environments, and Evaluation for Reinforcement Learning

Selection Process

For RLC 2025, we've updated our award process while keeping the core philosophy from last year. We believe in recognizing papers for their specific strengths, rather than for a single, overall score based on perceived novelty or impact. This means a paper can still win an award even if it has minor flaws in areas outside of its award-winning contribution. Our goal is to highlight papers that excel in one specific aspect, doing that one thing exceptionally well.

Based on feedback and new insights, we've made a few changes this year:

Author Eligibility: We've removed the rule that excluded papers with conference organizers as co-authors (with the exception of the awards co-chairs). We made this change to ensure that student first authors aren't unfairly penalized and to avoid negatively impacting researchers from underrepresented communities.
Reviewer Input: Instead of collecting nominations from reviewers, we placed more weight on the review scores to ensure award-winning papers had strong support from the reviewers.

To select the outstanding paper awards, we used the following selection criteria:

Methodological Rigor: The work must demonstrate outstanding rigor appropriate for its claims, such as error-free proofs for theory or robust, reproducible, and well-controlled experiments for empirical work.
Core RL Contribution: The paper must fundamentally advance Reinforcement Learning theory, algorithms, core concepts, or applications, rather than simply being inspired by RL concepts but using different techniques.
Clarity and Quality: The paper must be clear, well-written, and effectively communicate its claims, as well as the process and evidence in support of these claims.

Our selection process was a two-stage, independent effort between the two of us.

Stage 1: We independently reviewed the abstracts and meta-reviews of all 115 accepted papers. We then met to narrow down the pool to about 30 papers for a closer look.
Stage 2: We read the reviews for each of the 30 papers and looked at key aspects of the papers themselves. This led to a shortlist of 0-3 papers per category. We then rated each of these papers independently on a scale of 1 to 5. Although our individual ratings varied slightly, we were in complete agreement on the final award recipients.

This approach enabled us to recognize a more diverse range of papers, celebrating contributions that might have been overlooked by traditional review processes. We awarded papers with and without theoretical results, with simple and complex ideas, and with both small and large-scale experiments. This diversity is a major strength for our community.

This year, we awarded two papers for Scientific Understanding in RL but no awards for Pioneering Vision in RL or RL Contributions to Natural Sciences. For future RLC conferences, we'd love to see more submissions in these areas.

Congratulations to all the authors!

Description of Award Categories

Outstanding Paper Award on Applications of Reinforcement Learning

This award aims to acknowledge papers that demonstrate substantial progress on the application of reinforcement learning to complex, real-world problems. This award seeks to highlight groundbreaking work formulating real-world problems using the reinforcement learning framework, introducing a new application domain or challenge to reinforcement learning, or developing reinforcement learning methods that make significant progress on practical scenarios. The papers should display a notable level of practical utility and uphold a high standard of scientific rigor.

Outstanding Paper Award on Emerging Topics in Reinforcement Learning

This award acknowledges papers that make exceptional contributions to emerging topics in the field of reinforcement learning. This category seeks to recognize groundbreaking work on novel and forward-thinking ideas connecting RL with broader trends in machine learning that are poised to become central to future research and applications. Examples of such topics include, but are not limited to, foundation models, world models, and reinforcement learning from human feedback. The papers should demonstrate innovative approaches and the potential to significantly influence the direction of reinforcement learning research by opening up new avenues of exploration.

Outstanding Paper Award on Empirical Reinforcement Learning Research

This award recognizes papers that make significant contributions to the empirical aspects of reinforcement learning research. Examples include addressing fundamental practical challenges in reinforcement learning, introducing new empirical practices, methodologies, benchmarks, evaluation metrics, and visualization techniques, and providing tools and frameworks that will further enable empirical research. These papers should show a high standard of practical relevance and experimental rigor.

Outstanding Paper Award on Pioneering Vision in Reinforcement Learning

This award highlights papers that stand out with their forward-thinking vision and blue sky ideas in the field of reinforcement learning. The papers awarded in this category will present groundbreaking, visionary ideas, theories, or techniques in reinforcement learning, potentially reshaping current perspectives or opening new avenues for research and applications. The papers must demonstrate originality, creativity, and the potential to inspire transformative advancements in reinforcement learning.

Outstanding Paper Award on Resourcefulness in Reinforcement Learning

This award honors papers that demonstrate resourcefulness in empirical research. These are papers that overcome the high computational cost of empirical research in reinforcement learning in ingenious ways, promoting more frugal empirical research. Examples include showcasing original, cost-effective methodologies, and resource-efficient experimental designs. The papers should embody high standards of creativity and practicality without sacrificing experimental rigor.

Outstanding Paper Award on Reinforcement Learning Contributions to Natural Sciences

This award recognizes papers that make exceptional contributions to the advancement of natural sciences through reinforcement learning. Awarded papers will demonstrate how reinforcement learning methods have been effectively applied to generate new insights, drive discovery, or model complex phenomena in fields such as neuroscience, cognitive science, psychology, biology, animal learning, or related disciplines. These papers should exemplify scientific rigor, cross-disciplinary innovation, and a clear impact on our understanding of the natural world. They are expected to bridge reinforcement learning and natural sciences in a way that fosters new scientific methodologies or enhances existing ones.

Outstanding Paper Award on Scientific Understanding in Reinforcement Learning

This award celebrates papers that significantly advance scientific understanding in the domain of reinforcement learning. It encourages the development of well-founded, clear understanding of the behavior of existing algorithms or the nuances of different problem formulations or different environments. Awarded papers will fill gaps in our understanding of the field; they will bring clarity to unexplored aspects of existing algorithms, they will provide evidence to dispute common assumptions, or they will better justify common practices in the field. They should also demonstrate excellence in scientific rigor and clarity of exposition, with very well-defined claims.

Outstanding Paper Award on the Theory of Reinforcement Learning

This award acknowledges papers that provide exceptional theoretical contributions to the field of reinforcement learning. Examples include theoretical unifications, new theoretical frameworks or formalisms, mathematical models, results, and theoretical insights into existing RL practices. The papers must exhibit a high level of technical proficiency and innovation.

Outstanding Paper Award on Tooling, Environments, and Evaluation for Reinforcement Learning

Outstanding Paper Award on Tooling, Environments, and Evaluation for Reinforcement Learning Research recognizes papers that make significant contributions to support tools for reinforcement learning research. Examples include introducing new environments, datasets, benchmarks, evaluation metrics, visualization techniques, or frameworks, libraries, and tools that will further enable empirical research in reinforcement learning. These papers should show a high standard of practical relevance, accessibility, ease of use and reproducibility.