\documentclass[english,other]{jnlp_3.5}
\usepackage{jnlpbbl_2.2}
\usepackage[dvipdfmx]{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{url}
\usepackage{hangcaption_jnlp}
%%%%\usepackage{udline}
%%%%\setulminsep{1.2ex}{0.2ex}
%%%%\let\underline=\ul
\allowdisplaybreaks


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% author's definition
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\Volume{29}
\Number{1}
\Month{March}
\Year{2022}
%%%%\Issuetitle{Special Issue: XXXXXXXX}


\setcounter{page}{230}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\esubject{Society Column}
\etitle{ACL Rolling Review:\\ A New Format For Centralized Peer Review}
\eauthor{Raj Dabre\affiref{rd}} 

\headauthor{Dabre}
\headtitle{ACL Rolling Review}

\affilabel{rd}{}{National Institute of Information and Communication Technologies}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\maketitle


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

Central to the publication process for research papers is the concept of peer-review wherein researchers submit manuscripts to a publication venue (a conference or a journal), receive reviews, and if the reviews are favorable the manuscript is deemed to be officially accepted and published. This is true for the field of natural language processing (NLP) as it is for any other field. However, in recent years, the rapid rate at which NLP research has been proliferating has resulted in a strain on the traditional peer review system. Firstly, the number of manuscripts being submitted has exploded. This has increased the demand for reviewers, who are also often authors. Since, reviewing is an unpaid job in most cases, there is little incentive for researchers to write good quality reviews, other than altruism or a sense of wanting to write good reviews for others as they would like to receive good reviews themselves. Furthermore, reviewers often find themselves inundated with a backlog of manuscripts they have been tasked to review, and there is a natural tendency to decline to review when in this overloaded state. Simply increasing the number of reviewers is easier said than done, since reviewers are expected to have an in-depth knowledge of the field and this can only come with experience, which means that there is a risk of poor quality reviews. Secondly, the life of authors, who are also often reviewers, is equally burdened since reviews have a strong effect on their research roadmap and career plans. Authors have to commit to several deadlines spread throughout the year and given the unpredictable nature of research, they may be pressured to rush to submit manuscripts. If they are of poor quality, which is often the case with rushed manuscripts, the manuscript will likely get rejected and this also wastes reviewer time. Even if a paper is of good quality, it may get rejected either due to author-reviewer mismatches or due to the acceptance threshold, and thus the author will have no choice but to seek alternate venues, where the whole process begins again. Naturally, this is a major problem and recently, some short term changes were introduced\footnote{\url{https://www.aclweb.org/adminwiki/index.php?title=Short-Term_Reform_Proposals_for_ACL_Reviewing}} by the ACL (Association for Computational Linguistics) in an attempt to eliminate some of these problems. In short, these solutions involved (a.) findings papers, where papers that did not make it to the main conference were highlighted as papers of interest thereby giving authors some semblance of acceptance, (b.) reviewer awards, where reviewers who write good reviews receive awards which incentivizes high quality reviews, (c.) review mentoring, where experienced reviewers would create guidelines for junior reviewers, and (d.) the journal style revise-and-resubmit, where a paper rejected from one venue could be revised and submitted to another venue along with existing reviews in order to reduce the decision-making time. These measures seem to have had a reasonable impact on the overall process and in 2021, the ACL community decided to centralize this entire process under the moniker of ACL Rolling Review or ARR in short. ARR is divided into two fundamental steps: (a.) centralized rolling review, and (b.) submission to a publication venue. The overview is given in Figure~\ref{arr}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%% fig. 1
\begin{figure}[b]
\includegraphics{29-1ia22f1.pdf}
\hangcaption{Overview of ARR, which is divided into two loosely coupled phases: centralized rolling review (left) and publication venue submission (right).}
\label{arr}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Centralized Rolling Review}

In the centralized rolling review phase, authors submit their papers to a central pool of papers. The key difference compared to the previous approach for peer-review at *CL\footnote{This stands for any ACL recognized venues such as ACL, NAACL, EMNLP, EACL, etc.} conferences is that there are no pre-defined tracks and there is a monthly deadline for submissions. For example, authors could previously submit their papers to the morphology or machine translation track, but this is no longer the case, at least at this stage of ARR. While previous conferences required authors to submit by a fixed date 3--4 times a year, authors may now submit every month, currently the 15th of each month. Authors may edit or even retract their papers before the deadline. Furthermore, authors are given the option to make their anonymous manuscript publicly available. Up until now, authors would be in a rush to submit their works to arXiv a month before conference deadlines in order to ``claim'' their idea. By enabling authors to make their manuscripts publicly available anonymously, the expectation is that there will be lesser pressure on authors to rush and make an arXiv submission and enable them to focus on polishing their content. After the submission deadline, the usual procedures such as reviewer availability confirmation, desk rejections and initial paper assignment is done in about a week. This is where a ``new'' role comes into play, called as the action editor.

Different from the previous ways, area chairs are now replaced by action editors, a term that is perhaps well known to people who submit to journals. Action editors, which may form up to 20\% of the reviewer pool of 2000 to 4000 people, are responsible for managing reviewers where they can assign papers to reviewers, ensure that reviewers write reviews on time, analyze reviews, write a meta-review and then make recommendations to the senior area chairs (or senior action editors) regarding the fate of the paper. These recommendations, come in the form of private comments as to which venue the action editor thinks the paper should be published. Action editors may be tasked with managing 10 to 15 papers. Thanks to the automatic paper to reviewer matching system, papers are pre-assigned to reviewers that the automated system thinks are relevant. The action editor may choose to reassign papers if they deem it appropriate or if some reviewers decline to review or fail to review. After assignments are fixed, reviewers are given roughly 3 weeks to write, hopefully high quality, reviews. Action editors are expected to monitor reviewer actions and ensure that all reviewers hand in reviews by the deadline. Once the reviews are in, the action editors write their meta-reviews (summaries of the reviews) and make compliance checks. The compliance checks are minimal and involve ensuring that papers are up to standard with respect to relevance to the field, writing quality and formatting requirements. In the rare cases where compliance checks are not met, the authors may be prevented from resubmitting their papers for a particular duration to minimize paper spamming. This process is expected to take a little over a month, but given that ARR is in its experimental phase, this may need 6 to 7 weeks. Once the reviews and meta-reviews are available, the authors have a few choices available to them.

If the authors are satisfied with the reviews and scores, they may choose to proceed with the publication step. If not, they may choose to make revisions to their manuscript and resubmit it for another round of revisions. The authors can respond to reviews, enabling smoother interaction with reviewers. Ideally, the same action editor and reviewers will be involved, and there is a possibility that subsequent rounds after the first one may finish faster. It is also possible that the author-reviewer match is bad, in which case authors may request for a change in reviewers. After one or more of such rounds, it is expected that authors will be satisfied with the reviews and scores they have received. Should the authors be confident that their paper has a high likelihood of being accepted into a publication venue, they may choose to make their manuscript non-anonymously publicly available.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Submission To A Publication Venue}

A publication venue, such as any *CL conference, workshop or journal, will notify the community, say roughly two months in advance, that they will be accepting submissions, and that papers that have been reviewed (and satisfy the compliance criteria) can be submitted. There will be a publication submission period of roughly 1 month, and it is expected that the centralized rolling review deadline will be roughly 2 months before the publication submission deadline. The reason being that the centralized rolling review stage takes at least 1 month to get the first set of reviews. In other words, authors should submit their papers for review at least 2 months before the publication submission deadline. Once the publication submission deadline has passed, the senior area chairs and program chairs will sift through the submissions and choose papers to be presented in the conference based on a combination of numerical scores,\footnote{There has been a criticism of the numerical system in the past and this may be addressed as ARR evolves.} reviews and meta-reviews, private comments from authors and reviewers, optional discussion between the program chairs and the action editors, and consideration for diversity/direction of the program. Selected papers will be presented in the conference/workshop/journal. Authors of papers that are not selected can either submit them to the next publication opportunity, or can revise and resubmit the paper in an attempt to improve the reviews.

Readers may note that loosely coupling the reviewing and submission phase gives reviewers and authors significant flexibility in managing their submission workloads and review workloads, respectively. Before ARR, authors would probably be busy working on several papers right before a submission deadline, but now they can work on one paper at a time and hopefully have a less stressful experience. On the other hand, reviewers can express their availability to review for each monthly centralized rolling review cycle. As ARR is still new the authors and reviewers will probably experience the vestiges of tight deadlines, but once ARR is widely adopted this should resolve on its own.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Concerns}

While ARR is designed to address some existing concerns in the current conference peer-review process, there are several concerns as follows: 

\noindent \textbf{Handling blind review:} Naturally, during the centralized review stage, the papers are expected to be anonymous but during the publication venue submission stage papers authors may choose to reveal their names and affiliation in the paper, especially if they are satisfied with their reviews and scores.

\noindent \textbf{Handling areas and diversity in content:} During the centralized review stage, areas may be irrelevant as the major concern is in obtaining reviews and this stage is decoupled from any particular venue. Furthermore, if the program chairs of a venue wish to have a special theme track, they can provide authors with an option to submit papers to either the general pool or the pool for the theme. On the other hand, each venue has the right to define its own areas and authors can then submit to these areas during the publication venue submission stage. The program committee is usually tasked with ensuring scientific content diversity, and this can be dealt with in the call for submissions of reviews papers for publications.

\noindent \textbf{Handling the transition into ARR:} Since ARR may not be a perfect solution, venues may freely choose to partially or fully adopt it. This is not limited to the main conference but also to the workshops. Venues who choose the partial option can run their own reviewing and acceptance system and optionally accept a few papers from ARR. Since workshops tend to have a specific focus, they may tend to prefer partial adoption. Complete adoption would likely be more efficient/fair, but also require a decision from the ACL executive board to switch over to this new system and mandate this of the program chairs. One option would be to do a partial adoption for a few years, then move to a complete adoption. 

\noindent \textbf{Handling acceptance rates:} There are several organizations which measure the quality of conferences based on the acceptance rate, usually determined by the ratio of number of accepted papers to the number of submitted papers. This can be important for career advancement, so it is important to have an acceptance rate that appropriately represents the difficulty of publishing a paper in the conference. With ARR, this can be tricky because authors might submit their papers to the publication venue submission stage only if they are confident about the scores they have received. Therefore, acceptance rates may seem artificially high. Using the total number of papers submitted for reviewing may lead to an extremely low value. Two alternative strategies may work here, which focus on the fraction of authors/papers that prefer a particular venue or the median score of papers accepted to a venue. The first strategy involves asking authors to specify a desired venue during the publication venue submission stage and at the end of each year, statistics on the number of authors/papers preferring a particular venue can indicate its popularity and hence its acceptance rate. The second strategy involves computing the median score of papers accepted to a venue, and then determining the fraction of papers in the review pool that have the same score or higher. This fraction would then be an estimator of acceptance rates.  Instead of the median for each publication venue, we could also take the 10th or 20th percentile, representing the lower end of the venue's accepted range.

\noindent \textbf{Handling reviewer burnout:} There is a possibility that ARR may encourage authors to submit even more papers and this will lead to a burden on reviewers, which in turn will either lead to a lengthy review process and potentially poor quality reviews. While there is no way to control this, reviewers are free to indicate their availability to review each month in order to balance their reviewing load.

\noindent \textbf{Handling a potentially lengthy process:} The revise and resubmit option may encourage authors to submit incremental work over multiple rounds of review, which will consume substantial reviewer time. Placing a limit on the number of times revise and resubmit is allowed may be a solution, but sometimes authors may be forced to go through multiple iterations if they were unlucky enough to be assigned incompatible reviewers. In this case, authors will be provided with an option to make appeals to change their reviewers. Alternatively, some authors may be unlucky with having their highly rated papers being passed over repeatedly in favor of others, and this can be avoided by placing limits on how many times the publication of highly rated papers can be delayed. The centralized review system will enable better collection of such statistics, and this will help improve ARR in the future.

\noindent \textbf{Handling retractions:} Currently, retraction is not supported but is under consideration.

\noindent \textbf{Handling publication fees:} While most authors will choose to publish at conferences, the ARR may encourage submissions to edited journals like TACL which costs money. Currently, there is no system in place to handle a potential increase in publication fees, but it is under consideration.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}

In this paper, we have given an overview of the recently proposed ACL Rolling Review (ARR) system. ARR is expected to provide authors and reviewers with great flexibility in terms of submission deadlines, ensuring that revisions of papers are kept track of, enabling reviewer load balance and hopefully helping accelerate the publication process overall. We have explained the various steps involved in ARR in sufficient detail. In addition, we have noted potential issues and solutions to the same.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\acknowledgment

The author would like to thank the NLP community for their active participation and suggestions, which led to the concept of rolling reviews. The author is extremely grateful to Graham Neubig, Carnegie Mellon University, for his valuable feedback on the manuscript. The author acknowledges that most of the content in this paper was derived from the ARR proposal.\footnote{\url{https://www.aclweb.org/adminwiki/index.php?title=ACL_Rolling_Review_Proposal}}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{biography}

\bioauthor[:]{Raj Dabre}{%
obtained his Ph.D. from Kyoto University in March, 2018 and has been a researcher at the National Institute of Information and Communications Technology, Japan since May, 2018. His primary focus is on multilingual natural language generation (NLG) for low-resource languages, especially Indic languages. His other interests are in neural efficiency, speech recognition, image captioning, simultaneous translation and document translation.
}

\end{biography}


\end{document}



