Project AlexandriaA LAION research project

Open scientific knowledge

Scientific papers.
Structured knowledge.

We turn research papers into summaries and factual Knowledge Units that people can read, search and reuse. Then we test how much useful information survives.

See the results Read real summaries

A public collection today. An evaluated, open model pipeline for what comes next.

22M+structured paper summaries
in the existing public collection
18fields in each published
summary record
94.12%QA accuracy using our rank-128
summary adapter on 97 papers
The large collection and the new adapter benchmarks are separate releases.

01 / Two representations

Keep the findings.
Make the structure visible.

A summary helps a reader understand a paper. A Knowledge Unit makes individual facts and their relationships explicit. Alexandria explores both.

PAPER-LEVEL EXPLANATION

Structured summaries

A readable account organized into stable fields: the research question, methodology, key results, claims, limitations and more.

RESEARCH QUESTION
METHODS
CLAIMS + EVIDENCE

Useful for reading, literature comparison and field-specific retrieval.

FACTS AND CONNECTIONS

Knowledge Units

Structured records of a contextual description, entities, attributes and relationships. Extraction proceeds through source chunks instead of producing one whole-paper narrative.

ENTITYsample_A
tested_with
ENTITYmethod_B

Attributes stay attached to their entities.

Useful for granular fact retrieval and building graph-like representations. Diagram is schematic.

The Alexandria paper introduces the KU approach. Our newer full-paper experiments extend it with 500- and 1,000-word extraction chunks.

02 / How we evaluate

Can another model answer
questions from the representation?

We measure answerability: how well a fixed question-answering model can recover facts using a summary, Knowledge Units or the available original text.

  1. 1

    Freeze the questions

    The benchmark contains 97 papers and 970 four-choice questions—ten per paper. Questions were created from source text, before the tested representations were generated.

  2. 2

    Generate without the answers

    Summary and KU generators receive source text. They never see the test questions, gold answers, rationales or QA evidence annotations.

  3. 3

    Ask the same student

    The fixed student receives each question with one selected context. Accuracy is the number of correct answers divided by all 970 questions.

FIXED QA STUDENT / ANSWERER

Qwen2.5-7B-Instruct

This is the evaluation model. Gemma and Qwen 27B generate the representations; Qwen2.5 answers the questions.

Model card ↗
Question +No contextOriginal textKnowledge UnitsSummary→ same student → score
Protocol, uncertainty and what the score does not prove

The 97-paper set has 50 arXiv and 47 Bethgelab records. Its synthetic questions were authored by gpt-6-luna from source text. It is a length-filtered convenience sample; available text was not independently verified against complete publisher originals.

The student uses temperature 0.5, top-p 0.95, a 100-token output cap and frequency/presence penalties of 1.05. The historical answer prompt, ASCII sanitizer and parser are retained, with at most five formatting attempts. An answer that remains invalid is wrong. Failed generation keeps all ten questions for that paper in the denominator.

Confidence intervals resample whole papers, keeping each paper’s ten questions together. The answerer is stochastic, so unchanged texts can receive slightly different scores in another run. Scores measure answerability under this student and synthetic benchmark; they are not a human certification of factual correctness. Repeated benchmark use also limits independence.

Read the complete evaluation protocol · Download the frozen test set

03 / Shared benchmark

How much information
reaches the student?

97 papers / 970 questions

All five contexts below were freshly scored together on 9 October 2026. No context measures what the student can answer without paper information. Original text is the available-text reference.

Same papers, questions and answering model for every row
Context given to the studentCorrect answersQA accuracy95% paper interval
No paper contextQuestion and options only599 / 97061.75%58.14–65.26%
Full available paper textOriginal-text reference938 / 97096.70%95.57–97.84%
Knowledge Units · 1,000 wordsQwen 27B, source words per extraction call878 / 97090.52%88.66–92.37%
Knowledge Units · 500 wordsQwen 27B, source words per extraction call898 / 97092.58%90.72–94.33%
Structured summary · rank 128Gemma 4 12B IT · Qwen-distilled · merged FP8 · no thinking913 / 97094.12%92.37–95.77%
94.12% from a structured summary.

The Qwen-distilled Gemma rank-128 summaries score close to the 96.70% original-text reference in this run. Qwen’s 500-word KUs score 92.58%. These results describe one benchmark, not all scientific literature.

Source: matched 97-paper report · machine-readable numbers. The summary model uses merged FP8 weights and no thinking. KU chunk sizes count source words per call, not output words.

04 / Our open model adapters

Teach a 12B model
to write better summaries.

We trained LoRA adapters on top of Gemma 4 12B IT, using 865 source-only Qwen 27B generation outputs and their matching reasoning traces.

LoRA learns added, low-rank weight updates while retaining the base model. We trained ranks 64 and 128 for one epoch, specifically for summary generation.

The 97-paper external test set was excluded from this training corpus using identifiers, titles, source hashes and substantial-text overlap checks. The experiment trains generation; it does not train a separate quality-review or correction adapter.

865training papers128recommended rank1training epoch

What fine-tuning changed in the controlled comparison

Same prompt, sampling and thinking mode. These thinking-enabled runs precede the optimized, no-thinking deployment in the main table.

97 papers / 970 questions · one epoch for each adapter
Summary generatorCorrectQA accuracy95% intervalFailed papers
Base Gemma · no adapter850 / 97087.63%85.26–89.90%0 / 97
Summary LoRA · rank 64853 / 97087.94%82.68–92.68%7 / 97
Summary LoRA · rank 128897 / 97092.47%89.07–95.15%2 / 97

Source: matched training evaluation. Failures remain in the denominator. Summary lengths differ, so this is not a fixed-length comparison. Merging, quantization and batching were tested in a separate deployment experiment.

RECOMMENDED SUMMARY ADAPTER

Gemma 4 12B IT · Qwen-distilled · rank 128

An adapter to load with the Gemma 4 12B IT base. The model card documents training, inference and evaluation.

Get the LoRA adapter ↗
Also released:Qwen teacher · rank 64Ornith teacher · rank 128Ornith teacher · rank 64

05 / Knowledge Unit adapters

A different output.
A separate held-out test.

We also trained rank-128 KU adapters. These learned from 391 Qwen KU completions from the original 97 papers, so we test their generalization on 20 different papers and 200 frozen questions.

This 10 October follow-up freshly scores both chunk sizes. Its questions were authored by Qwen 27B and automatically checked against source evidence by Qwen2.5. This cohort is separate from the 970-question benchmark above.

Disjoint 20-paper test · each cell includes correct / 200 and a paper-bootstrap interval
KU generator1,000-word source chunks500-word source chunks
Qwen 27B · teacher88.00%176 / 200 · 95% CI 85.00–91.50%89.50%179 / 200 · 95% CI 87.00–92.00%
Gemma 4 E4B IT · base64.50%129 / 200 · 95% CI 57.00–72.00%75.50%151 / 200 · 95% CI 65.50–83.00%
Gemma 4 E4B IT · KU rank 12880.00%160 / 200 · 95% CI 73.00–86.00%82.50%165 / 200 · 95% CI 71.50–91.00%
Gemma 4 12B IT · base63.50%127 / 200 · 95% CI 54.00–71.50%75.00%150 / 200 · 95% CI 64.50–84.00%
Gemma 4 12B IT · KU rank 12885.00%170 / 200 · 95% CI 80.50–89.00%88.50%177 / 200 · 95% CI 84.00–92.00%
No context 44.50%Original text 91.00%Summary rank 128 89.00%

With 500-word chunks, the Gemma 12B KU adapter reaches 88.50%, compared with 75.00% for its base model. The earlier summary adapter reaches 89.00% on these same questions. Twenty papers give wide uncertainty intervals; do not pool these scores with the 97-paper results.

Source: full 20-paper comparison, including failures and paired intervals · Gemma 12B KU adapter · read complete KU examples

06 / Inside a summary

Eighteen fields.
More than one way to ask.

The published dataset has 18 LLM-generated summary fields, alongside source metadata and OpenAlex enrichment. Choose a field to see what it captures and how it can be searched.

EVIDENCE

Claims & evidence

claims

Individual scientific claims with supporting evidence, contradicting evidence and implications. This lets a search target a claim rather than an entire paper.

ILLUSTRATIVE SEARCH QUESTION

Find claims about a mechanism together with contradictory evidence.

This panel explains the schema. The real dataset and saved examples are linked below.

Schema detail: our newer evidence-grounded benchmark uses 19 top-level keys, adding an influential-citations section and short source-evidence snippets. Field names and storage formats differ slightly from the public 18-field collection.

07 / What this makes possible

Search for the part
of a paper you actually need.

Stable fields let a retrieval system search claims, methods, results or limitations directly, then return the parent paper and its supporting context.

AN EXAMPLE OF FIELD-AWARE RETRIEVAL

“Which papers use this method—and what limitations do they report?”

methodological_detailscontradictions_limitations
Illustrative query, not a live search result.
01

BM25: match the words

Find exact terminology, named techniques and phrases inside a selected field.

02

Vector search: match the meaning

Retrieve semantically related claims or methods even when their wording differs.

03

Keep evidence and provenance

Return the source paper, field and available evidence so a reader can inspect the result.

The public collection already documents BM25 and FAISS search artifacts. We want to expand coverage and make these structured representations more useful, while continuing to measure their quality.

08 / Better paraphrasing

Preserve the facts.
Reduce copied wording.

Fluent summaries can still reproduce passages from their source. We tested backtranslation and direct Gemma 4B editing as a repair stage after generation, measuring both the remaining copied wording and the information available to the QA student.

OUR PARAPHRASING TARGET

At most five consecutive source words in narrative prose. A match of six or more words fails this strict target. The audit compares each summary with its own full available source; it reports narrative prose, evidence quotations and bibliographic metadata separately.

This is an engineering rule, not a legal copyright threshold. An overlap can be a conventional scientific phrase, so the detector identifies wording to inspect rather than deciding whether a passage is improper.

Two ways to rewrite a flagged passage

TRANSLATE AND RETURN

TranslateGemma 4B
backtranslation

We take a flagged English passage with nearby sentence context, translate it into German, and translate the German back into English. The round trip can change wording and sentence structure while retaining the original information.

The reference uses greedy decoding. The sampled experiment uses temperature 0.35 and top-p 0.95, with either one attempt or up to two. A second attempt is made only after the first is rejected; it starts again from the original English rather than translating an already modified passage.

A sampled candidate is accepted only if both translations finish, the protected signatures match, its longest source match is at most five words, and the entailment check passes in both directions. Otherwise the original passage stays.

TranslateGemma model card ↗
EDIT IN ENGLISH

Gemma 4 E4B IT
direct repair

A small instruction model rewrites the selected English passage directly. It receives a short context window around the copying, with instructions to preserve the scientific meaning, rather than generating a new whole-paper summary.

We tested one pass and an iterative policy with up to 20 attempts, thinking disabled, and a temperature schedule of 0.35, 0.65 and 0.95. Candidates are checked against the immutable pre-repair passage.

This policy can retain safe partial progress: copied-word coverage must decrease without increasing the longest match, protected signatures must remain unchanged, and both entailment checks must pass. Unresolved matches remain visible in the audit.

Gemma repair model card ↗

Three different roles: Gemma 4 12B with the rank-128 summary adapter generates the initial text. TranslateGemma 4B or Gemma 4 E4B IT repairs selected passages. Qwen2.5-7B-Instruct answers the evaluation questions. The repair models do not receive the questions or gold answers.

What happens between generation and evaluation?

  1. 01

    Scan the source

    Find contiguous matches of at least six normalized whitespace words. Keep separate summary statements separate, and audit against untruncated available source text.

  2. 02

    Keep the context

    Select the affected sentence window with preceding context. Edit narrative prose; preserve literal evidence quotations and bibliography, and audit them separately.

  3. 03

    Check each rewrite

    Compare numeric, unit and formula signatures with the original window. A separate entailment model must score at least 0.90 in both directions.

  4. 04

    Check the assembled text

    Validate the structure, protected fields, document signatures and longest matches after inserting edits. Revert unsafe outputs and keep them in the results.

  5. 05

    Measure both outcomes

    Re-audit the saved summary and ask the fixed QA student the same frozen questions. QA scores never choose an edit, a retry budget or which papers to retain.

MATCHED REPAIR COMPARISON · 12 OCTOBER 2026

Less copying. How much information survives?

97 papers / 970 questions per row

Every row starts from the same saved Gemma 12B rank-128 FP8 summaries, generated without thinking. All six conditions were QA-scored together in this run. Greedy backtranslation and direct Gemma rows reuse frozen repaired texts; the one- and two-attempt sampled backtranslation texts were created for this experiment.

Same original summaries, papers, questions and QA student · all 970 answers retained per condition
Repair applied to the summaryQA accuracyCopied-word coverageQA change vs original
Percentage points · paired 95% CI		Strict ≤5-word pass
Original summary · no repairGemma 12B · rank 128 · merged FP8 · no thinking94.12%913 / 970 correct15.23%Unrepaired reference0.00 ppReference0 / 97
Backtranslation · greedyFrozen TranslateGemma 4B reference93.81%910 / 970 correct9.69%36.4% relative reduction−0.31 pp95% CI −1.03 to +0.310 / 97
Backtranslation · one sampled attemptTranslateGemma 4B · temperature 0.3593.40%906 / 970 correct9.60%36.9% relative reduction−0.72 pp95% CI −1.55 to +0.000 / 97
Backtranslation · up to two attemptsA second independent attempt only after rejection93.09%903 / 970 correct9.09%40.3% relative reduction−1.03 pp95% CI −1.86 to −0.210 / 97
Direct Gemma repair · one passGemma 4 E4B IT · English editing93.81%910 / 970 correct12.59%17.3% relative reduction−0.31 pp95% CI −1.03 to +0.410 / 97
Direct Gemma repair · iterativeGemma 4 E4B IT · up to 20 attempts93.81%910 / 970 correct7.71%49.3% relative reduction−0.31 pp95% CI −1.13 to +0.520 / 97

What “coverage” means

For each summary, count the narrative words covered by any source match of six or more words, counting each word once. Divide by its narrative word count, then average across the 97 summaries. Lower is better. The reduction underneath is relative to the original summary’s 15.23% coverage.

What “strict pass” means

A whole summary passes only when no narrative passage contains a source match longer than five words. 7.71% copied-word coverage does not mean only 7.71% of papers fail. Every tested summary still has a long narrative match: strict passes are 0/97 in every row.

WHAT WE LEARNED
  • Iterative direct Gemma repair gives the largest reduction in this comparison: copied-word coverage falls by 49.3%, from 15.23% to 7.71%. Its QA score is 93.81%, versus 94.12% for the original summaries. The paired interval includes zero, so this run does not establish either a quality loss or equivalence.
  • A second sampled backtranslation attempt reduces coverage further but also lowers QA: 9.60% → 9.09% coverage, and 93.40% → 93.09% QA. The two-attempt result is 1.03 percentage points below the original, with a paired interval of −1.86 to −0.21. More attempts are not an automatic improvement.
  • The strict target remains unsolved. Rejecting a rewrite preserves the original wording. Protected quotations also remain unchanged: 34/97 summaries have an evidence match of at least seven words, separately from the narrative metrics above. These methods reduce copying; they do not certify complete paraphrasing.
Exact settings, numeric checks and the wider 1,000-version repair test

The direct editor runs in BF16 with thinking disabled, up to 128 active sequences, length-sorted prompts, prefix caching and chunked prefill. Its context limit is 4,096 tokens and its output cap is 768 tokens. The sampled translator runs in BF16 with batches up to 512, a 2,048-token context limit and a 768-token output cap. A declared retry budget is a ceiling, not a promise that every passage is rewritten that many times.

The entailment filter is DeBERTa-v3-base-mnli-fever-anli. It checks whether the original supports the candidate and whether the candidate supports the original, each at a 0.90 threshold. Numeric, unit and formula signatures provide an additional conservative filter. Retained repairable outputs have zero detected critical-signature drift, but these checks cannot verify every mathematical expression or prove scientific equivalence.

The wider direct-Gemma stress test processed 1,000 summary versions from 962 unique papers: 865 Qwen teacher drafts, 97 FP8 evaluation summaries and 38 BF16 versions of evaluation papers. One pre-existing unparseable input was retained, leaving 999 auditable versions. These groups are not pooled into the 97-paper QA table.

Across both direct-Gemma conditions, whole-document guards reverted 33 output variants covering 19 distinct summary-version identifiers. Reverted texts remain in the results. No papers are dropped to improve the score, and QA never selects a repair. Evidence quotations and metadata retain their original wording; their overlap is separately reported rather than silently exempted from the audit.

Model revisions, seeds, acceptance policies, paired intervals and exact coverage values are available in the machine-readable benchmark data. Five words is a project convention; it is not a guarantee of copyright clearance, scientific correctness or independent human approval.

Earlier QA run: why the same saved repair can have a slightly different score

The initial direct-Gemma experiment also freshly evaluated these frozen contexts on 12 October. Its scores are shown below as a separate run. Qwen2.5 uses temperature 0.5, so a fresh evaluation of an unchanged text can differ by a few answers. In the six-condition comparison above, identical contexts reuse the first answer set from that run; distinct contexts receive fresh QA calls.

Earlier direct-repair run · same 97 papers / 970 questions · separate from the six-condition table
ContextQA accuracy95% accuracy intervalQA change vs original
Percentage points · paired 95% CI
Original summary94.12%913 / 970 correct92.47–95.67%0.00 ppReference
Direct Gemma · one pass94.12%913 / 970 correct92.47–95.67%+0.00 pp95% CI −0.52 to +0.52
Direct Gemma · iterative93.92%911 / 970 correct92.16–95.57%−0.21 pp95% CI −0.93 to +0.52
Greedy backtranslation93.51%907 / 970 correct91.75–95.15%−0.62 pp95% CI −1.24 to −0.10

The copying measurements are unchanged because the saved summaries are unchanged. Repeated scores are not additional independent papers and are not averaged into a stronger quality claim. The 97-paper benchmark has been used repeatedly; independent validation and human review remain necessary to establish broader reliability.

A SEPARATE APPROACH · PROCEDURAL REPAIR V3

Local rewording without a generative editor

This variant uses context-checked substitutions, typed phrase rules and descriptor reordering. It protects numbers, formulas, logical qualifiers, names and quotations, and reverts unsafe edits. It reduces mean narrative copied-word coverage by 39.3%, with no generative model call in the repair stage.

15.23% before9.25% after

In its separate 13 October QA run: 94.23% before → 94.12% after. The change is −0.10 percentage points; the paired 95% interval is −0.72 to +0.62. Strict narrative passes remain 0/97.

This row comes from a different fresh QA run and is not merged into the repair table above. Audit and QA numbers ↗

Repair benchmark data: curated scores, copying audits and model settings · read five saved backtranslated summaries. The example reader is curated and includes residual overlap flags; it is not a representative repair success estimate.

09 / Explore the collection

Twenty-two million summaries.
A foundation to build on.

The existing LAION Scientific-Summaries release contains roughly 22 million structured paper summaries. It was generated with Gemini 2.0 Flash Lite; the Gemma adapter evaluations on this page describe our newer generation pipeline.

We plan to summarize as many additional papers as feasible, expand scientific coverage and improve paraphrasing. Open adapters, inspectable examples and transparent evaluation are part of that work.

Open Scientific-Summaries ↗Browse dataset files ↗

Read five complete summaries

Explore saved Gemma rank-128 outputs after guarded backtranslation, with numerical notation and provenance.

Open the example reader →

Try the summary adapter

Get the Qwen-distilled rank-128 LoRA and follow its model card to reproduce generation on Gemma 12B.

Open the model repository →

Inspect and contribute

Read the Alexandria paper, investigate the code and benchmark artifacts, and help improve the next iteration.

Open the Alexandria repository →

The five-summary reader is a curated cross-domain sample, not a representative quality estimate. It preserves actual saved outputs, including residual copying flags.