Assigning Papers to Referees (original) (raw)

The Conference Paper-Reviewer Assignment Problem

Decision Sciences, 1999

The Conference Paper Assignment Problem (CPAP) is the problem of assigning reviewers to conference paper submissions in a manner intended to minimize whingeing. It is assumed that papers are reviewed by members of a preset program committee (PC), each of whom has the opportunity to bid on papers prior to the assignment algorithm being run. In this survey, we show that CPAP is in P if the only information given is individual program committee members' preferences for individual papers. However, if both preferences and expertise (based on, say, keywords) are given, the problem is potentially more complex.

An algorithm for automatic assignment of reviewers to papers

Scientometrics, 2020

The assignment of reviewers to papers is one of the most important and challenging tasks in organizing scientific conferences and a peer review process in general. It is a typical example of an optimization task where limited resources (reviewers) should be assigned to a number of consumers (papers), so that every paper should be evaluated by highly competent, in its subject domain, reviewers while maintaining a workload balancing of the reviewers. This article suggests a heuristic algorithm for automatic assignment of reviewers to papers that achieves accuracy of about 98-99% in comparison to the maximum-weighted matching (the most accurate) algorithms, but has better time complexity of Θ(n 2). The algorithm provides an uniform distribution of papers to reviewers (i.e. all reviewers evaluate roughly the same number of papers); guarantees that if there is at least one reviewer competent to evaluate a paper, then the paper will have a reviewer assigned to it; and allows iterative and interactive execution that could further increase accuracy and enables subsequent reassignments. Both accuracy and time complexity are experimentally confirmed by performing a large number of experiments and proper statistical analyses. Although it is initially designed to assign reviewers to papers, the algorithm is universal and could be successfully implemented in other subject domains, where assignment or matching is necessary. For example: assigning resources to consumers, tasks to persons, matching men and women on dating web sites, grouping documents in digital libraries and others.

An automated conflict of interest based greedy approach for conference paper assignment system

An automated conflict of interest based greedy approach for conference paper assignment system, 2020

Reviewer Assignment Problem (RAP) is a crucial problem for the conference due to time constraints and inadequate availability of expert reviewers. A fair evaluation of paper is key to an author's success, paper quality, conference reputation, and productive usage of funds. Recent studies reflect on the issue of reviewer bias in bids favoring authors belonging to the top institution and higher authority. Existing Conference Management Systems (CMS) are solely dependent upon self-declared Conflict of Interest (CoI) made by the authors, and reviewers. In literature, existing studies considers topic similarity, potential CoI, and reviewer's workload as trivial factors for ensuring review quality. Other factors include the diversity and authority of a reviewer. Past studies propose several theoretical optimization models. In this paper, we first individually model the factors using the best possible strategy in a constrained-based optimization framework. We tried to propose a completely novel framework that can be practically implemented to improve upon the performance of existing CMS. We map the RAP to an equilibrium multi-job assignment problem. Moreover, we propose a meta-heuristic greedy solution to solve it using weighted matrix factorization. We redefine an assignment quality metric required to validate such assignments. A real conference assignment data set collected from EasyChair is used for a comparative study. The TPMS is used as a baseline because it also uses similar factors, and due to its integration with widely used Microsoft CMS. The results show that the mean assignment quality of the proposed method is superior to other benchmark RAP systems.

Modelling the conference paper assignment problem

2020

In this paper 4 we describe different constraints and models for the conference paper assignment problem. While the core problem is a simple flow problem, additional constraints often arise to tailor a solution to specific wishes, or to increase perceived fairness for reviewers and/or submissions. We show some results from actual conferences paper assignments, and also investigate scalability of the method for large-scale events.

Decision Support System for Assignment of Conference Papers to Reviewers

Computational Collective Intelligence, 2019

Research conferences are held to share research progress and novel findings among scientists and encourage the growth of academic communities. Assigning papers to reviewers is the most critical and arduous task for conference organizers. Usually, conference committees must distribute hundreds of publications to reviewers in a short time. Moreover, other restrictions-such as a limited number of reviewers and avoiding conflict of interest-make the process of wholly distributed more difficult. Manually choosing reviewers can be an unfair and time-consuming process. In this study, we developed a support system for automatically assigning submitted papers to reviewers. For this, we extracted features from reviewer profiles and their networks to compute their quality and relevance with respect to the submitted papers. Experiments on a DBLP dataset showed that our system achieved more accurate results than other methods.

A decision support tool using Order Weighted Averaging for conference review assignment

Pattern Recognition Letters

Assigning papers to reviewers is a large, long and difficult task for conference chairs and scientific committees. The paper reviewer assignment problem is a multi-agent problem which requires understanding reviewer expertise and paper topics for the matching process. This paper proposes to elaborate on some features used to compute reviewer expertise and aggregate multiple factors to find the fittest combination of reviewers for each paper. Expertise information is gathered implicitly from publicly available information and a reviewer profile is generated automatically. An OWA (Ordered Weighted Average) aggregation function is used to summarize information coming from different sources and rank candidate reviewers for each paper. General constraints for the RAP (Reviewer Assignment Problem) have been incorporated into a real case example: (i) conflicts of interest between a reviewer and authors should be avoided, (ii) each paper must have a minimum number of reviewers, and (iii) each reviewer load cannot exceed a certain number of papers.

Matching Papers and Reviewers at Large Conferences

2022

Peer-reviewed conferences, the main publication venues in CS, rely critically on matching highly qualified reviewers for each paper. Because of the growing scale of these conferences, the tight timelines on which they operate, and a recent surge in explicitly dishonest behavior, there is now no alternative to performing this matching in an automated way. This paper studies a novel reviewer– paper matching approach that was recently deployed in the 35th AAAI Conference on Artificial Intelligence (AAAI 2021), and has since been adopted by other conferences including AAAI 2022 and ICML 2022. This approach has three main elements: (1) collecting and processing input data to identify problematic matches and generate reviewer–paper scores; (2) formulating and solving an optimization problem to find good reviewer–paper matchings; and (3) the introduction of a novel, two-phase reviewing process that shifted reviewing resources away from papers likely to be rejected and towards papers closer...

Designing of Fair and Accurate Reviewer Assignment in Peer Review

2020

One of the normal issues looked in allocating recommendations submitted to the conferences, journal publications and so forth is the assignment of the proposition to the suitable reviewers. This is likewise named as the reviewer assignment issue. Here, where the skill level of a reviewer who is engaged with reviewing a proposal ought to be streamlined to ensure the choice of the best reasonable proposal. Picking a suitable reviewer includes the mastery as well as thinks about assorted variety and irreconcilable circumstances among them. Toward this path, various arrangements have been given by researcher previously, yet a portion of the issues continues in this field. The proposed system features a review on the current strategies proposed by the analysts to tackle the reviewer assignment issue and different keyholes in the current situation alongside the potential arrangements. Keywords— Information Retrieval, Journals, Conferences, Reviewer Assignment

Designing of Fair Reviewer Assignment in Peer Review

2020

Assigning Papers to Referees (original) (raw)

Related papers