elvis· @omarsar0 · X·· 2 小时前AI 评分62
AI 导读
DAIR.AI 推荐了一篇将 Jev 等决策模型接入智能体 harness 的论文。作者从 Jev 在智能体自身状态上的回答蒸馏出开放的 4B 决策模型,在该智能体内表现超过 Jev 且运行成本更低。JevAgentHarness 把每个常规选择交给决策模型,仅在置信度超过按智能体自身轨迹拟合的阈值时才采用其答案,否则仍由 LLM 决定;使用拟合阈值时工具与参数级联使成本降低 14.2%,500 个任务结果均未改变,而固定 0.5 阈值则损失 21.5 个点。他们还用智能体犯错后到达的状态、以 Jev 答案为标签训练 AgentDecision-4B,在不同于训练数据的任务上该智能体解决率 91.5%,Jev 为 82.0%;在熟悉任务上这类 4B 模型让智能体成本降低 81.1% 至 89.0%、最高快 2.5 倍,成功率差距在 3 个点以内。
正文
Recommended paper integrating decision models like Jev into harnesses.
Must-read paper integrating decision models like Jev into harnesses. They distilled an open 4B decision model, mostly from Jev's answers on the agent's own states, that beats Jev inside that agent and is much cheaper to run. Their harness, JevAgentHarness, sends each routine choice to the decision model and uses its answer only when the confidence clears a threshold fitted on the agent's own trajectories. Below the threshold, the LLM decides as usual. With the fitted threshold, tool and argument cascades cut cost by 14.2% without changing the outcome of any of 500 tasks. A fixed threshold of 0.5 lost 21.5 points. They also train AgentDecision-4B on states the agent reaches after its own mistakes, with Jev's answers as labels. On tasks unlike its training data, the agent solved 91.5% with it and 82.0% with Jev. On familiar tasks, 4B models trained this way made agents 81.1 to 89.0% cheaper and up to 2.5 times faster while staying within 3 points of success. Paper: https://academy.dair.ai/papers/jevagentharness-benchmarking-deploying-and-training-decision-models-that-share-c-2610.21412在 X 查看被引用的帖子
来源:elvis · x.com