作者开源基于 LangGraph、Celery 和 MCP 的容器化多智能体研究流水线

I wanted to share the architecture behind a project I recently open-sourced: a Multi-Agent Research & Report Generator.

My goal was to build a production-grade system that takes a complex user prompt, breaks it down into structured sub-topics, autonomously gathers literature, critiques its own findings, and compiles a polished Markdown report.

Moving an agentic LLM workflow from a linear Python script into a robust, distributed web application required some heavy lifting. Here is a breakdown of the architecture and how the components in the repository fit together.

1. The AI Engine & Agent Graph

At its core, the brain of the app runs on LangGraph as a four-stage state machine:

* Planner: Breaks the core question down into 3–5 specific sub-topics.

* Researcher: Pulls data for each sub-topic using external tools.

* Critic: Evaluates the draft on coverage, accuracy, and depth (scoring 0–5). If the score is below 3, it triggers a conditional loop back to the Researcher with explicit feedback injected into the prompt (capped at 3 retries).

* Writer: Converts accepted research notes into a formatted executive summary.

1. Protocol-Isolated Tooling (MCP)

To keep the agents modular, I didn't hardcode API scrapers into the agent logic. Instead, I implemented a custom Model Context Protocol (MCP) server running over stdio. This safely exposes tools like web_search (DuckDuckGo), arxiv_search, and semantic_scholar_search to the Researcher agent as a standardized tool-calling interface.

1. Distributed Microservices Architecture

Running high-latency LLM loops synchronously is a recipe for HTTP timeouts. I completely decoupled task ingestion from background execution:

* FastAPI Gateway: Accepts the user prompt and immediately returns a task_id with a 202 Accepted status. No hanging HTTP connections.

* Redis Broker & Celery Workers: The FastAPI gateway pushes jobs into a Redis queue. A horizontally scaled cluster of Celery workers (--scale worker=4) continuously pulls tasks from the broker, spinning up isolated LangGraph execution loops in the background.

1. Durable State Checkpointing

In-memory state management is too fragile for a distributed pipeline. I configured state snapshots and thread recovery asynchronously using LangGraph's AsyncPostgresSaver backed by PostgreSQL. This ensures that multi-turn conversational context and tool outputs survive service restarts and enable granular run recovery.

1. The Frontend & Telemetry

For the user interface, I built a clean Streamlit frontend. Once the Celery workers finish processing the LangGraph pipeline, the frontend parses the final JSON payload from the backend, renders the formatted Markdown report, and displays agent telemetry metrics (like the number of iteration loops taken and the final critic score) before offering a .md download.

TL;DR

I built an asynchronous, containerized multi-agent research platform using LangGraph, FastAPI, Celery, Redis, PostgreSQL, and Streamlit. It utilizes a critic-gated retry loop to minimize hallucinations, uses MCP for tool execution, and handles heavy concurrency via a distributed Celery queue. Everything is containerized cleanly on Docker Compose.

Link to the repo: https://github.com/theallkeeeymist/multi-agent-researcher-report-generator

I'd love your feedback!

This has been a massive learning experience in scaling AI workflows, but I know there is always room to improve. What else can I do to make this architecture even more robust for production?

Are there alternative patterns for handling the queuing, better ways to manage the LangGraph state across workers, or specific failure modes I should be planning for? I'd love to hear your thoughts!

submitted by /u/TheKingAlchemist
[link] [留言]