用作者自己的代码和超参数复现论文数值偏低，该如何在论文中呈现？
Couldn't reproduce a published paper's numbers with its own code and settings. How do I present this? [R]
一位研究者用原作者发布的代码和论文中的超参数复现某已发表方法，多数 benchmark 结果在噪声范围内一致，但两个 benchmark 上跨三个随机种子低了好几个点，且相隔数月重复四次结果相同。
I'm writing a machine learning paper. One of my baselines is a published method. I reran it with the authors' released code and the exact hyperparameters listed in their paper. On most benchmarks my numbers match theirs within noise, but on two benchmarks my reproduction is clearly lower, by several points, over three seeds. I've repeated the reproduction four different times, months apart, with the same result.
I plan to report both: their published numbers and my reproduction with the exact configuration, clearly labeled. My worry is that reviewers tend to trust the published number more than a submission's reproduction, and may assume I ran the baseline incorrectly.
Questions:
What makes a reproduction convincing to reviewers? So far I have: the authors' code unchanged, their hyperparameters, matching results on the other benchmarks as a sanity check, and multiple seeds.
Should I contact the authors or open a GitHub issue first, and should I mention that in the paper?
How do I word it so it doesn't read as an attack on the other paper, but my results also don't look weak next to the published numbers?
Have reviewers accepted "published vs. reproduced" rows in a main table for you, or did they end up in the appendix?
submitted by /u/InsectActive95
[link] [留言]
来源：r/MachineLearning · reddit.com