在线智能体评测太贵，AgentVQA 这类离线基准能否替代？
Anyone else tired of how expensive agent evaluations are? [D]
有开发者抱怨在真实环境中运行通用智能体评测耗时耗算力、结果还难以复现，转而使用离线基准 AgentVQA 来比较不同 VLM 的视觉理解能力。该基准无需搭建环境即可离线运行，并声称数据比其他含大量歧义样本的基准更干净。他因此提问：这类离线基准能否减少对在线评测的依赖。
So I have been working on some general use agents that work with different domains but these require online simulations to test for some of the domains. And running agents in actual environments takes so much time and compute and then reproducing results is another headache. My goal is to test which VLMs can be better in general for agents irrespective of a singular domain.
I found there’s a benchmark called AgentVQA that tests how well models understand visual information without having to run these expensive full agent evaluations. And they do it all offline without setting up any kind of environments and also claim to have much cleaner data compared to other benchmarks available which have so many ambiguous samples.
Do you guys think offline benchmarks like this could actually help reduce the need for online evals? I personally found it quite useful for the work I’m doing and it’s much cheaper and easier with this. I want to know what others think and if this might solve their problems.
Here’s the benchmark: https://zlab-princeton.github.io/AgentVQA/
submitted by /u/Key_Recording9766
[link] [留言]
来源：r/MachineLearning · reddit.com