How to Build a Scientific Literature-Review Agent Without Getting Rate-Limited

Sarah Choy·Published May 3, 2026·10 min read

Build a literature-review agent on raw arXiv + PubMed + Semantic Scholar today and you'll hit 429s before you finish ten papers. Here's why the rate limits got worse, what PaperQA / Undermind actually do under the hood, and a working pattern that survives a real review session.

TL;DR

• •arXiv began throttling aggressively in late 2024 after LLM scrapers triggered 429 storms; the documented limit is 1 request every 3 seconds, but real-world burst tolerance is lower.
• •Semantic Scholar's 1,000 req/sec is shared across all unauthenticated callers — effectively unusable for production agents.
• •PubMed E-utils caps unauthenticated traffic at 3 req/sec; with an API key, 10 req/sec.
• •Working agents (PaperQA2, Undermind, Elicit) all use a proxy / cache layer in front of these sources.
• •API Pick Academic Search wraps arXiv, PubMed, bioRxiv, and medRxiv in one rate-managed endpoint — 5 credits per call, only-on-success.

The problem in one paragraph

Free academic APIs are wonderful and they're being eaten alive. arXiv, PubMed, and Semantic Scholar were designed in an era when 'one researcher writes a script that polls every few seconds' was the worst-case load. Now every undergraduate with Python writes an LLM agent that fans out fifty parallel calls to read one paper's references. Multiply by thousands of similar agents and you get the pattern arXiv staff describe on X as the "arXiv-pocalypse" of late 2024 — sustained Rate Exceeded responses, degraded service, and a wave of new throttling rules that hit indie builders harder than the bad actors.

The result: a literature-review agent that works on five papers in dev environment falls over halfway through a real review session. The user gets a half-finished answer with three citations instead of fifteen.

This article is about how to build the agent so that doesn't happen.

What's actually rate-limited and where

What working products actually do

PaperQA2, Undermind, Elicit, ResearchRabbit, and the agentic RAG papers (e.g. Open-Source Agentic Hybrid RAG Framework, arXiv 2508.05660) all converge on a similar pattern. Three engineering moves matter most:

1. Cache by DOI / arXiv ID, not by query

The same paper gets requested by many different queries. Caching at the search-result level barely helps; caching at the paper-identifier level does. A small Redis (or even SQLite) layer keyed on doi:10.1038/... pays itself back in one afternoon of agent traffic.

2. Treat the citation graph as the slow-changing index

Semantic Scholar's strength isn't keyword search; it's the citation graph. PaperQA2 uses it to expand from one seed paper to a related cluster, not to discover papers from scratch. That's a much smaller request volume — one call per paper, not hundreds — and stays well under the rate limit.

3. Accept latency as the trade-off, OR proxy the rate limit

Either you make users wait 30-60 seconds for a careful, throttle-respecting query (PaperQA2's choice), or you add a layer that aggregates traffic across users and presents a single burst-tolerant interface to the agent (the API Pick choice). Both work. Mixing them — a low-latency UX over rate-limited public APIs without a buffer — is what fails.

Working code: a literature agent that finishes

The minimum viable agent that survives a real review session:

import requests
from anthropic import Anthropic

KEY = "pk_yourkey"
client = Anthropic()

def fetch_tool(path: str) -> dict:
    return requests.get(f"https://www.apipick.com{path}/tool-schema").json()["claude"]

# Two tools: academic search + URL extract for the full body
TOOLS = [
    fetch_tool("/api/search/academic"),
    fetch_tool("/api/extract"),
]

SYSTEM = """You are a literature-review research assistant. Process:

1. Use academic_search with the user's question to find relevant papers
   from arXiv, PubMed, bioRxiv, and medRxiv. One call usually returns
   the right set of seed papers.

2. For papers worth citing in detail, use extract_urls on the paper's URL
   to pull the body. Batch up to 5 URLs per call.

3. Answer the user's question with inline citations in the form
   [Author Year, doi/arxiv URL]. Quote findings verbatim where possible.

4. If the search returned <3 substantive results, expand the query and
   try once more. Don't loop indefinitely — say "I couldn't find a strong
   answer in the indexed literature" if nothing matches.

Be precise. Cite every factual claim. Distinguish preprints (arXiv,
bioRxiv, medRxiv) from peer-reviewed (PubMed) when it matters."""

def call_tool(block):
    name_to_path = {"academic_search": "/api/search/academic", "extract_urls": "/api/extract"}
    r = requests.post(
        f"https://www.apipick.com{name_to_path[block.name]}",
        json=block.input,
        headers={"x-api-key": KEY},
        timeout=60,
    )
    return {"type": "tool_result", "tool_use_id": block.id,
            "content": r.text, "is_error": r.status_code != 200}

def review(question: str) -> str:
    msgs = [{"role": "user", "content": question}]
    while True:
        r = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=4096,
            system=SYSTEM,
            tools=TOOLS,
            messages=msgs,
        )
        msgs.append({"role": "assistant", "content": r.content})
        if r.stop_reason == "end_turn":
            return "\n".join(b.text for b in r.content if b.type == "text")
        if r.stop_reason == "tool_use":
            results = [call_tool(b) for b in r.content if b.type == "tool_use"]
            msgs.append({"role": "user", "content": results})

print(review("Recent advances in LLM-based protein structure prediction (2025)"))

Cost per question:

• 1-2 academic_search calls (5-10 credits = $0.005-$0.010)
• 1 extract call covering 3-5 papers (6-10 credits = $0.006-$0.010)
• ~5,000 input + 1,200 output tokens to Claude (~$0.04)

Round figure: ~5 cents per literature-review answer with citations. At 100 questions/day that's $5/day — about what one reasonable researcher's daily caffeine costs.

Sub-vertical tweaks

Biomedical literature

Bias the agent toward PubMed (peer-reviewed) and explicitly mark bioRxiv/medRxiv hits as preprints. When asking "what does the latest evidence say", the agent should weight peer-reviewed higher. A line in the system prompt — "If preprint and peer-reviewed sources disagree, defer to peer-reviewed and note the discrepancy" — handles this cleanly.

Mathematical / CS

arXiv dominates here. The signal-to-noise ratio is better than in biomed, and citations matter more than recency for foundational work. Search with broader queries and let the agent prune.

Drug discovery / clinical

Pair with Clinical Search (ClinicalTrials.gov + FDA labels + ChEMBL + DrugBank) for the regulatory and bioactivity dimensions that academic search can't cover. The combination — peer-reviewed literature + trial registry + structural data — is how a drug-repurposing or pharmacovigilance agent gets useful.

Where this generalises

The rate-limit problem is not unique to academic search. Any open public dataset that shipped APIs in the pre-LLM era is being subjected to the same load patterns now: SEC EDGAR, USPTO, EPO, public-records databases, weather services, government open data. The same engineering pattern works for all of them — cache at the entity level, layer a managed proxy, accept latency as the alternative when you can't afford the proxy.

The deeper shift: in 2026 it's no longer reasonable to expect a free public API to absorb LLM-agent load directly. The endpoints aren't disappearing — but they're getting wrapped by a layer of paid managed services that exists to translate "polite human researcher" rate limits into "burst-tolerant agent traffic". API Pick Academic Search is one of those layers for the literature-review use case. Expect more.