ICLR 评审评分从 1-10 改为 1-4 引发争议：量化噪声与并列分数问题

I was venting about the ICLR 1-10 to 1-4 change and Gemini wrote this (unprompted), which I thought was pretty good.

Compressing a continuous perception of quality into four coarse bins creates massive discretization noise. From a measurement theory standpoint, stripping out quantization levels inflates rounding error and severely degrades the signal-to-noise ratio:

1. Discretization Error and Quantization Noise

If true paper quality $X$ is modeled on an underlying latent continuum, mapping it to a coarse integer grid introduces a quantization error $e = Q(X) - X$.

Under uniform quantization with step size $\Delta$, the variance of the quantization noise is approximately: $$\sigma_q^2 = \frac{\Delta^2}{12}$$
Moving from a 10-point scale ($\Delta_{10} \approx 1$) to a 4-point scale over the same dynamic range ($\Delta_4 \approx 3.33$) increases the variance of the rounding error by a factor of roughly: $$\left(\frac{\Delta_4}{\Delta_{10}}\right)^2 \approx 11\times$$

When an Area Chair averages $N=3$ or $N=4$ reviews, the sample mean $\bar{Y} = \bar{X} + \bar{e}$ carries a significantly larger error component. Coarse binning actively destroys fine-grained ordinal ranking before aggregation even begins.

2. Loss of Variance-Stabilizing Nuance

Reviewers naturally hesitate to use the extreme anchors (1 and 4), collapsing almost the entire distribution into a binary coin toss:

The "Forced Choice" Trap: A paper that is a standard, unexciting solid poster (a classic 6 on a 10-point scale) now forces the reviewer to pick between 2 (Weak Reject) and 3 (Weak Accept). A coin-flip choice between a positive and negative recommendation has enormous leverage.
Loss of the 8 vs. 6 Distinction: On a 10-point scale, an enthusiastic champion giving an 8 can mathematically lift a skeptical 5 into an acceptable range ($6.5$ average). On a 4-point scale, that same champion gives a 4 while the skeptic gives a 2. The average is $3.0$—identical to two lukewarm reviewers giving $3$ and $3$. Strong positive signal is completely flattened.

3. Ties and the Combinatorial Floor

With $N=3$ reviewers on a 1–4 scale, there are only 20 possible multiset outcomes (from $\{1,1,1\}$ to $\{4,4,4\}$), yielding just 10 possible mean values:

$$\{1.00, 1.33, 1.67, 2.00, 2.33, 2.67, 3.00, 3.33, 3.67, 4.00\}$$

At typical acceptance rates (~25–30%), hundreds or thousands of papers inevitably pile up at the exact same tie points:

A score of 2.67 (one 3, two 2s or one 4, one 2, one 2—e.g., $\{3,3,2\}$ or $\{4,2,2\}$).
A score of 3.00 (e.g., $\{3,3,3\}$ or $\{4,3,2\}$).

When mass clusters form on identical discrete averages, the score ceases to function as a ranking metric.

What Program Chairs Think They Are Fixing vs. Reality

The Theoretical Justification	The Actual Practical Consequence
Eliminate subjective calibration: Prevents "harsh" vs. "generous" reviewers from debating whether a paper is a 5, 6, or 7.	Amplifies boundary sensitivity: The jump from 2 to 3 carries an existential swing that dwarfs calibration differences.
Kill the "neutral" cop-out: Eliminates the non-committal middle score (like a 5 or 6).	Forces artificial polarization: Reviewers with ambivalent but valid critiques must choose between endorsing or killing the work.
Shift weight to qualitative text: Forces ACs to read arguments rather than sorting an Excel spreadsheet by mean score.	Increases AC workload and variance: When scores tie across hundreds of papers, final decisions hinge entirely on the idiosyncratic taste of a single AC.

Removing precision from an already noisy measurement instrument does not magically clean the underlying signal—it just forces the downstream aggregator to make unprincipled tie-breaks.

submitted by /u/valuat
[link] [留言]