August 9-12, 2024
Amherst, MA

Review Process for RLC 2024

Why this matters

The quality of a conference depends on the quality of the work published there. One of the most important things we can do is put in place structures that make it more feasible to identify incorrectness in submitted papers and avoid rejecting good work.

The goal of this proposal is to improve the quality of reviews. This benefits authors, who get better feedback, and also benefits RLC by improving the quality of papers.

One issue is that the community has grown very quickly. This growth has increased the burden on senior researchers, making it less feasible to devote time to reviewing and to training the next generation of reviewers. Junior researchers do not get the same opportunity to develop their reviewing abilities. Further, junior reviewers may have a harder time appreciating the importance of the work, especially if it looks different than the papers they are used to, yet are unfairly tasked with assessing novelty.

Proposal for the review process

A key criteria for the RLC is stated as follows. “The RLC peer review process prioritizes rigorous methodology over perceived importance, aiming to foster scholarly discussions on both well-established and emerging topics in RL.” The below review process reflects this principle. Note: It is the opinion of the PC that focusing the review process on correctness will not result in an increased acceptance rate compared with the current review process.

Another fundamental principle is to make the review process as lightweight as possible. Extra work just detracts from spending time reading the paper carefully and writing a good review. The trend for improvements to reviewing has been to add more work to improve reviewing; the goal of this proposal is to reduce workload.

High-level overview

Motivation for two reviews

There may be concerns that two reviews is insufficient, due to noise in the process. This structure for reviewing, however, should already be lower noise. The reasons are as follows.

It is important to recognize that adding more reviews does not necessarily decrease variance. Adding more reviews changes the distribution, and this distribution might actually have higher variance (e.g., we add less reliable reviewers). Even worse, this distribution might have high bias—we want to get closer to the distribution of expert reviewers, not a broad distribution over senior researchers, students, bloggers, and others that the current review pools are built from. Also note that for journal papers it is common for an Action Editor to make decisions based only on two reviews.

More details about the review process

Other details about the implementation