Large Language Models (LLMs) are adept at rapidly analyzing and summarizing written content like news articles and blog posts. Scientific papers are a different story.
Accurately distilling such work requires faithfulness to specific methodological details and technical knowledge, making it harder for AI to summarize. Frontier LLMs like Claude Opus can produce quality summaries of scientific papers, but API dependencies and computational cost make them impractical for literature review at scale. Making this frontier AI summarization more affordable, accurate, and scalable could deliver significant cost and security advantages for government and research institutions that need to process large volumes of technical literature.
In pursuit of this goal, NSPDI researchers created this paper showing how to improve scientific summarization from Claude Opus 4.6 into LLaMA 3.1 8B (LoRA) and LLaMA 3.3 70B (QLoRA) using 1,500 arXiv papers on CS, physics, and mathematics. Because our distilled model can run locally on a single consumer-grade GPU without requiring an internet connection, agencies can leverage our methods in air-gapped and security-sensitive settings where sharing data with external AI services is prohibited.
Key Takeaways:
- Frontier AI summarization can be more affordable and deployable at scale. Using our methods to summarize 100,000 technical papers via the Claude API would cost roughly $2,000 at current batch rates for Claude Opus 4.6 ($5/$25 per million input/output tokens standard, halved under the 50% batch discount), given our average token counts.
- Standard AI evaluation tools give misleading accuracy grades, as several widely used automated metrics rank summaries that copy text verbatim higher than those producing better quality synthesis of information.
- Policymakers should also be skeptical of benchmark claims, as systems that appear to underperform on standard benchmarks may actually be producing higher-quality, more readable output. This indicates human review remains essential for high-stakes analysis.