用学校历年试卷生成考题：LLM 出题流水线求二次意见
Generating exam questions from a school’s own past papers: looking for a second opinion on the pipeline [D]
一位开发者正在为高中平台构建题库功能，用 LLM 将学校历年试卷和评分标准 PDF 拆分为子题并按课程目标打标，再由小模型生成新题、另一模型盲答校验，选择题答案在代码中对照评分标准核验，目前约 80% 的选择题通过全部检查。
Hi all,
I’m building a question bank feature for a school platform (high school, mixed Cambridge + in-house curriculum). The idea: new questions are generated from the school’s own past papers. The curriculum’s learning objectives, which are already in our database, act as a guard rail so questions stay in scope. A curriculum admin approves everything; teachers only use the approved bank.
I have a working prototype:
- Question papers + mark schemes (PDFs) go to an LLM, which splits them into items at sub-question level and tags each to objectives
- A small model writes new questions; a different model answers them blind as a check
- Multiple-choice answer keys are checked in code against the mark scheme, not trusted to the model
- So far: ~80% of MCQs pass all checks. Structured/practical questions are weaker; the small model tends to invent wrong chemistry in lab set-ups, and a mid-size model fixed most of it
What I’d love input on:
- PDF extraction. The PDFs’ text layer mangles science notation (Δ dropped, subscripts split, table columns interleaved), so right now I send the whole PDF to the LLM. Has anyone used docling, marker or similar on exam papers and got genuinely more precise results, especially for equations, tables and diagrams?
- Retrieval. Objective tags give exact retrieval for “past questions on topic X”. Is pgvector worth adding from day one, or only once the bank is large enough to need semantic de-duplication?
- Cost at scale. Experience with batch APIs or small models at thousands of questions? Where did quality fall off?
- Quality measurement. Beyond human review, how do you judge whether AI-generated assessment items are actually good? Any eval approaches that held up?
Stack: NestJS + Postgres (schema per tenant) on a self-hosted VPS. Exam content stays on our server; only curriculum text and question drafts go to the model.
If you’ve worked on something similar (edtech, document AI, assessment generation), I’d really appreciate comments, or a DM if you’re up for a longer chat. Happy to share more of the architecture there. Also open to paid consulting if it’s a good fit.
Thanks!
submitted by /u/SnooOpinions9858
[link] [留言]
来源：r/MachineLearning · reddit.com