shad0w：为本地 LLM 分类结果做缓存的小型查表工具

I built a tiny table that learns your local LLM's classification answers and serves the repeats in microseconds [D]

AI 导读

作者发布 shad0w，它位于任意 OpenAI 兼容服务前（Ollama 开箱可用），记录模型回答，累计约 1,000 条后编译出一张小表，在 CPU 上以微秒级响应重复问题。该表只在与模型在留出样本上一致率 95% 以上时才作答，其余仍交给模型；项目以 Apache-2.0 开源，使用 Python/JS/C 编写。

正文

project: https://2796gaurav.github.io/shad0w/

If you use a local model for classification, you know the wait: seconds per message, for questions it has answered hundreds of times. shad0w sits in front of any OpenAI-compatible server (Ollama works out of the box: llm="ollama/llama3.1" or `shad0w proxy --upstream http://localhost:11434/v1`), logs the model's answers, and after ~1,000 of them compiles a small table that answers the repeats on CPU in microseconds. It only answers where it agrees with your model 95%+ of the time on held-out answers; everything else still goes to the model.

Fun part: it learned Snake from an LLM's moves and keeps playing at 10 moves a second where the LLM can't: https://2796gaurav.github.io/shad0w/arena/

Apache-2.0, Python/JS/C. Happy to answer anything, and I'd love to hear where it breaks.

submitted by /u/gauravc2796
[link] [留言]