PC members and, optionally, external reviewers can rate one another’s reviews. We hope this feedback will help reviewers improve the quality of their reviews. The interface appears above each visible review:

When rating a review, please consider its value for both the program committee and the authors. Helpful reviews are specific, clear, technically focused, and, when possible, provide direction for the authors’ future work. The rating options are:

Average

The review has acceptable quality. This is the default, and should be used for most reviews.

Very helpful

Great review. Thorough, clear, constructive, and gives good ideas for next steps.

Too short

The review is incomplete or too terse.

Too vague

The review’s arguments are weak, mushy, or otherwise technically unconvincing.

Too narrow

The review’s perspective seems limited; for instance, it might overly privilege the reviewer’s own work.

Not constructive

The review’s tone is unnecessarily aggressive or gives little useful direction.

Not correct

The review misunderstands the paper.

HotCRP reports the numbers of non-average ratings for each review. It does not report who gave the ratings, and it never shows rating counts to authors.

To find which of your reviews might need work, simply search for “rate:−”. To find all reviews with positive ratings, search for “re:any rate:+”. You may also search for reviews with specific ratings; for instance, search for “rate:helpful”.

Chairs set how ratings work on the review settings page.

A review’s ratings are visible to any unconflicted PC members who can see the review, but HotCRP tries to hide ratings from review authors if they could figure out who assigned the rating: if only one PC member could rate a review, then that PC member’s rating is hidden from the review author.