Summarising a long PDF: are the facts right?
Task: Summarise a 40-page report for a busy manager without dropping key numbers or inventing any.
A summary that reads well can still be wrong. We check every summary against a list of facts from the report, and against claims the report never makes.
This is a test plan. Nothing has been run yet, so there are no scores or verdicts below.
How we tested
- Every tool gets the same PDF and the same prompt.
- A person reads the report and writes an answer key: 15–20 facts a good summary must include (numbers, names, conclusions), plus 5 tempting claims the report does not make.
- Fact recall is computed automatically: the share of facts found, minus a penalty for each wrong claim that appears.
- A person also scores coverage (are the most important points there?) and readability, from the raw output only.
- Cost and time are recorded for each run.
What we scored
- Facts right · objective · weight 3
- Share of the answer-key facts present, minus 25% for each claim the report never made.
- Covers what matters · human-judged · weight 1
- Are the three or four most important points there?
- Easy to read · human-judged · weight 1
- Could a busy manager act on it in two minutes?
Inputs every tool received
The report (PDF)Placeholder
Placeholder: PDF to be provided.
Show the exact prompt
Summarise this report for a busy manager in under 400 words. Include the key numbers exactly as the report gives them, and say what the report concludes. Do not add information that is not in the report.
Inputs still needed
- A real 30–50 page PDF with numbers and tables that we may share publicly (for example a public annual report or government study).
- The answer key: 15–20 required facts and 5 claims the report does not make, written by a person who read the report.
- Optionally an Arabic report for a second round.