机器学习研究中通过问卷做人工评估是否常见？
Is human evaluation through surveys common in ML research?[R][D]
一名硕士生用微调模型与 Qwen/OpenAI API 对同一批图像生成反馈，导师建议用 Google Form 让参与者比较哪个更准确有用，收集约 60 份回复并做统计分析。他询问这种人工评估在机器学习研究中是否常见、60 人样本量是否合理，以及如何处理乱答或不认真读题的参与者。
Hi,
I am a master's student working on a project where our fine-tuned AI model provides feedback on images. We are also comparing its feedback with outputs from Qwen/OpenAI APIs on the same images.
My postdoc and PhD mentors suggested creating a Google Form where participants compare the feedback and select which one is more accurate or useful. They asked me to collect around 60 responses and perform statistical analysis to determine which model performs better.
I would like to know:
- Is this type of human evaluation common in machine learning research?
- Is a sample size of around 60 participants reasonable?
- How do researchers deal with participants who answer randomly or do not read the questions carefully?
- If reviewer feedback later requires additional evaluation, is it common to conduct another survey?
I would appreciate any advice on possible issue I should discuss with my PostDoc before proceeding.
Thank you.
submitted by /u/Dapper-Perspective21
[link] [留言]
来源：r/MachineLearning · reddit.com