If you ask an LLM:

Distill the most important information from these sources. What are the key figures and key points?

you are asking a million-dollar question that sounds simple and is not. The model will answer with confident bullets. The honest question underneath is:

How does it know—or decide—which pieces count as “most important”?

This note fathoms that decision under the hood, then turns the map into operator technique. It is not a product review of any chat app.

The through-line: the model will not invent your values; you encode them as predicates, schemas, and pipelines.

---

One-sentence crystal

An LLM does not possess human judgment of stakes, novelty, or consequence; when it “finds key points,” it ranks and regenerates text by statistical patterns—attention, frequency, position, rhetorical scaffolding, similarity to your prompt, and what instruction-tuning rewarded as a “good summary”—then presents that ranking in fluent language.

---

The uncomfortable split: human importance vs model “importance”

LLMs do not have human values or an understanding of real-world stakes. That is not an insult; it is architecture. They predict and (when aligned) prefer outputs that look like helpful, structured answers—not because they feel the cost of a missed dosage or a buried legal clause.

So when you say “most important,” the model must operationalize that phrase with proxies. Part I names those proxies and the mechanisms under them. Part II is what you do once you accept the proxies are real.

---

Part I — Under the hood: proxies for “importance”

What the machine is actually optimizing

Before heuristics, the stack:

  your prompt + source text
           │
           ▼
  tokenization  →  embeddings / hidden states
           │
           ▼
  attention + feed-forward layers  (what can “see” what)
           │
           ▼
  next-token (or decoding) distribution
           │
           ▼
  instruction-tuned preferences  (“be a good summarizer”)
           │
           ▼
  fluent bullets that *look* like key points

Two different jobs get conflated in product language:

A model can attend to the right paragraph and still smooth the claim into conventional wisdom. Or it can miss the paragraph and still write a confident abstractive summary. Fluency is not fidelity.

---

Proxy 1 — Repetition (frequency and “aboutness”)

If an author returns to a phrase or concept across many paragraphs, training statistics and attention patterns often treat that concept as central. Crude intuition: if ~30% of the document orbits “supply chain bottlenecks,” a summarizer is more likely to put bottlenecks in the bullet list.

Why the proxy works: Human writing often is about what it repeats. Theses get restated; jargon anchors a field.

The flaw: A paradigm-shifting sentence once can outrank a repeated platitude for a human reader—and lose for the model. Single-line bombs (“we invalidate assumption A”) die without frequency or structural fireworks.

Under the hood (extra depth): “Repetition” is not a simple counter in the transformer. Related ideas reappear as nearby embeddings, entity chains, and attention sinks toward recurring tokens. Still, rare critical claims remain underweighted unless you mark them in the prompt (“prioritize claims that contradict prior work even if mentioned once”).

---

Proxy 2 — Position (primacy, recency, lost-in-the-middle)

Models and long-context use show a stubborn pattern: beginnings and endings often get more effective attention than the middle of a long pack—the lost-in-the-middle problem. Training data and document conventions reinforce the habit: introductions state theses; conclusions restate findings.

Why the proxy works: Many documents are written that way. Academic abstracts and executive summaries train the “good summary” prior.

The flaw: The decisive table on page 47 of 100 loses to a bland sentence on page 1. Counter-intuitive methods buried mid-paper disappear.

Under the hood: This is not only “bias in the weights.” It interacts with how you stuff the context window: order of chunks, truncation, and whether retrieval ever surfaces page 47. If your pipeline never places the critical span near the query or near the edges of the window, no amount of “be thorough” in the system prompt fully fixes it.

Operator move: Chunk with overlap; retrieve by embedding and by section title; ask explicitly for “findings not only in intro/conclusion”; put user question after sources or use formats that re-anchor attention.

---

Proxy 3 — Structural and linguistic cues (rhetorical scaffolding)

Models are strong at the furniture of argument:

• Signposts: “In conclusion,” “Crucially,” “The primary finding is,” “Most importantly,” “We contribute…”
• Discourse roles: problem → method → result → limitation
• Formatting residue after PDF parse: headings, lists, bold—if the parser preserved them

Why the proxy works: Authors signal importance on purpose. Catching the signal is rational.

The flaw:

• Marketing copy abuses the same signposts.
• A dry standards document may hide the load-bearing rule in a numbered subclause with zero “crucially.”
• PDF extraction can destroy heading hierarchy; the model then sees a flat token soup and leans harder on frequency and position.

Under the hood: Instruction-tuned models have seen millions of “Summary / Key takeaways” pairs. They learn genre: short parallel bullets, nominalizations, hedge words. That genre can be filled even when the source is thin—hallucinated structure is still structure.

---

Proxy 4 — Semantic similarity to the prompt (attention as a spotlight)

Bare instruction: “distill the most important information” → the model falls back on generic summary priors.

Add a word: “distill the most important financial information” → attention and residual pathways bias toward spans that co-activate with finance-ish tokens (revenue, margin, risk, …).

This is the practical heart of “how it decides”:

Importance becomes relevance to the query embedding / instruction, not relevance to the universe.

Why the proxy works: You usually do have a standpoint. “Key” for a CFO ≠ “key” for a firmware engineer.

The flaw: Over-narrow prompts delete cross-cutting risks (“safety,” “legal,” “reproducibility”) that never used your keyword. Over-broad prompts revive frequency and position.

Under the hood (attention, without the mysticism): In transformers, attention scores are a learned function of queries and keys: roughly, which earlier tokens should influence the next prediction. When the instruction says “financial,” representations of that instruction help select source spans that look related in representation space. That is not a spreadsheet of business criticality. It is geometric nearness + training.

Multi-head attention can track several patterns at once (entities, numbers, negation)—but heads are not labeled “human stakes” or “regulatory duty of care.”

---

Proxy 5 — The “average human” prior (training data gravity)

Pretraining on a vast web of human text teaches what people usually extract from genres: dosages and side effects from medical-ish text; benchmarks and ablations from ML papers; conflict and quotes from news.

Why the proxy works: Defaults match median reader expectations.

The flaw: Contrarian, unorthodox, or early-warning arguments get smoothed toward the mean. The model has seen more conventional takes than lonely correct ones. Summaries can sound balanced while sanding off the author’s actual thesis.

Under the hood: Post-training (instruction tuning, preference optimization) amplifies “helpful assistant” summaries: tidy, multi-sided, slightly generic. That can raise readability and lower fidelity to a sharp minority claim.

---

Proxy 6 — What the user material left out (and what practitioners must add)

The five proxies above already explain most chat behavior. A full fathom needs more:

6a. Abstractive compression, not extractive truth

Many systems rewrite rather than quote. Key-point lists may merge two sentences into a third that never appears—fluent, wrong, or overconfident. “Important” becomes “what fits in N bullets,” not “what is entailed by the source.”

6b. Instruction-following as a costume

“List three key points” is a format constraint. The model will often produce exactly three, even if the document has one load-bearing claim or twelve. Cardinality in the prompt is a decision procedure.

6c. Chunking and RAG (when “sources” are big)

If the pipeline is retrieve-then-read:

1. Retrieval decides the candidate pool (embedding similarity, BM25, metadata filters).
2. Generation decides the wording from that pool.

A fact never retrieved is “unimportant” by pipeline, not by philosophy. “The LLM ignored page 47” is sometimes “the retriever never gave page 47 to the LLM.”

6d. Decoding and sampling

Temperature, nucleus sampling, and beam search change which high-probability summary surface form you see. Importance is not a single stored ranking; it is sampled from a cloud of plausible summaries.

6e. Evaluation mismatch

Automatic metrics (ROUGE, etc.) reward n-gram overlap with a reference summary, not moral or operational importance. A system can score well while missing the one number that would change a decision.

---

A worked micro-scenario

Suppose a 80-page toolchain reliability report contains:

Ask generically for “key points,” and you often get green CI + observability future work. Ask for “silent failures, rates, and misconfigurations,” and attention shifts—you supplied the value function the model lacks.

That is the hinge into Part II: if the model only has proxies, your job is to write a better query, not to hope for better vibes.

---

What “importance” is not (myths)

---

Part II — Force what actually matters: the query-engine stance

To force an LLM toward what you care about, stop treating it like a human reader who “gets the vibe of the paper.” Treat it like a query engine over a corpus: narrow predicates, grounded rows, adversarial views, and scoped ranges.

You are not making the model grow judgment. You are externalizing judgment into the query plan.

The single biggest mistake: saying “summarize”

When you ask for a summary, you often trigger the model’s default programming:

• surface-level extraction
• conventional wisdom
• heavy reliance on introduction and conclusion
• tidy bullet genre that looks complete

“Summarize” is a soft instruction. Soft instructions get soft proxies (Part I).

Ban “summary” as the main verb when the stakes are real. Replace it with targeted extraction against an explicit schema of what may leave the document.

---

Technique 1 — Banish “summary” for targeted extraction

Give narrow scan criteria. Force the model to ignore the framing the author set up for casual readers.

Extraction prompts that rewrite the value function

The contrarian prompt

Extract only the claims, data points, or arguments in this document that challenge conventional wisdom. Do not include standard background information.

Hits: Proxy 1 (frequency of platitudes) and Proxy 5 (average-human prior). You explicitly reward rare or against-the-grain spans.

The tension prompt

Identify any points in the text where the author expresses uncertainty, highlights limitations in their own argument, or glosses over edge cases.

Hits: Authors hide load-bearing doubt in hedges. Summaries often drop limitations; this query puts them in the WHERE clause.

The actionable / mid-document prompt

Ignore the introduction and conclusion entirely. Read the methodology and results, and extract only operational data or step-by-step mechanisms.

Hits: Proxy 2 (primacy/recency). You forbid the cheap pages and force attention onto the middle meat—if those sections are in context.

The decision prompt (from Part I, sharpened)

I must decide [go/no-go decision]. Extract only facts that change that decision. If a sentence does not change the decision, omit it.

Hits: Importance becomes decision delta, not “what a Wikipedia lead would say.”

---

Technique 2 — Quote first, analyze second

Abstractive models love to flatten rhetoric into sterile AI-speak and to smooth unorthodox arguments into conventional ones (Proxies 5 and 6a).

Rule: every insight must be entailed by a visible span before commentary is allowed.

Prompt pattern:

For every key insight you identify, you must first provide a direct, verbatim quote from the text that supports it. Only after the exact quote may you explain its significance in one or two sentences. If you cannot quote it, you may not claim it.

Why it works:

Optional hard mode: require section / page / heading when the pipeline provides them.

This is the chat equivalent of SELECT span FROM document WHERE … before JOIN interpretation.

---

Technique 3 — Assign an adversarial persona

With no role, the model defaults to a polite generic assistant that often agrees with the text and produces harmonious takeaways (alignment + summary genre).

Roles are not magic theater. They are instruction tokens that reweight which patterns count as a successful answer.

Combine with quote-first: skeptic claims without quotes are fanfic.

Caution: Personas can overfit tone (everything becomes a risk). Pair with a second pass: “Which of these risks are explicitly supported by quotes vs inferred?”

---

Technique 4 — Defeat lost-in-the-middle on purpose

If you pour a long PDF into one broad question, the middle often loses (Proxy 2). “Be thorough” is not a substitute for range restrictions.

Bracket the query (page / section scope)

Analyze only pages 15–25 (or only the section titled “Methodology”). Identify …

The model cannot attend to tokens that never enter the window—but even inside the window, explicit range reduces the temptation to answer from the abstract.

Sequential processing (chat as a cursor)

Do not ask for everything at once:

We will analyze this document section by section. First, read only the “Methodology” chapter and extract …
Next, only “Results” …
Then reconcile contradictions across sections.

You turn one giant SELECT * into a cursor walk—closer to how careful humans actually read.

The “needle” command

Pay special attention to the middle of this document. Scan for any mention of [specific topic] and extract every sentence related to it. Prefer recall over brevity.

“Every sentence” fights the three-bullet costume (Proxy 6b). Specific topic fights generic summary.

Pipeline note

In RAG systems, bracketing is often metadata filters (section=methodology) or forced chunk IDs, not vibes. The chat phrasing above is the same idea with fewer moving parts.

---

Technique 5 — Compose the full query plan

Adversarial persona + targeted extraction + verbatim quotes is the minimum stack. A durable plan looks like this:

1. Scope     → which pages/sections are legal to use
2. Role      → skeptic / risk / domain specialist (optional but sharp)
3. Predicate → contrarian | tension | decision-delta | mechanism-only
4. Grounding → quote-first (or quote-only table)
5. Format    → table or schema, not free “summary”
6. Second pass → rank / decide / what did we miss (human or second query)

Example composite prompt

Act as a critical peer reviewer for an engineering reliability report.
Scope: ignore abstract, introduction, and conclusion; use only sections “Failure modes” and “Incident notes” (or pages 20–48 if paginated).
Task: extract claims about silent failure, misconfiguration, or rate of wrong greens.
For each claim: (1) verbatim quote, (2) one-line significance, (3) whether evidence is measured, inferred, or anecdotal.
Do not produce a general summary. If nothing matches, say so and quote the nearest near-miss.

That is a query, not a plea for wisdom.

---

Technique 6 — Pipeline moves (when chat is not enough)

Part I’s engineering checklist, tightened:

Part II turns soft “importance” into a query. Parts III–IV turn messy PDF reality into rows and schemas the query can hit.

---

Part III — Strict structure: messy text → JSON / CSV

Narrative freedom is the enemy of pipelines. To turn broken PDF text into pristine CSV or JSON, strip the model’s essay impulse and force rule-bound parsing.

Raw PDF paste is usually noise: broken lines, missing delimiters, OCR artifacts, inconsistent spacing, multi-column scramble. Prompts that say “put this in JSON” without a schema invite invention (Proxy 5 + abstractive merge).

Link to Part I–II: a schema is a hard predicate. null rules defeat filler. Zero-chatter defeats summary costume.

The 5 rules for structural prompts

1. Define the exact output schema

Never: “put this into JSON/CSV.”
Always: keys/columns, types, order, delimiter.

• JSON: target skeleton or TypeScript-style interface; arrays vs objects explicit.
• CSV: header list in order; delimiter (comma vs tab); quoting rules.

2. Set normalization and cleaning rules

Dirty values become downstream bugs. Specify transforms at extract time:

3. Handle missing or ambiguous data

Models love inventing values or writing "Unknown". Lock behavior:

• JSON: if missing or not certain → null (do not guess).
• CSV: leave the cell empty; do not write N/A / None unless you defined that as the sentinel.

4. Enforce zero-chatter (code-only output)

Block “Sure, here is your JSON:”.

Output ONLY valid JSON inside a single ```json fence. No introduction, notes, or closing prose.

Same for ```csv. Then your parser can split on the fence and JSON.parse without regex archaeology.

5. Provide a few-shot dirty→clean example

When multi-column text is interleaved or OCR is brutal, show one short example of messy input mapped to the desired object/row. Few-shot steers format better than adjectives (“be careful”).

---

Template A — Messy PDF text → clean JSON

You are a data extraction parser. Extract structured data from messy PDF text.

### Target JSON Schema
Return an array of objects matching this exact structure:
[
  {
    "invoice_number": string or null,
    "date": "YYYY-MM-DD" or null,
    "vendor_name": string or null,
    "items": [
      {
        "description": string,
        "quantity": integer,
        "unit_price": float,
        "total": float
      }
    ],
    "grand_total": float or null
  }
]

### Data Normalization Rules
1. Clean currency values ("$1,200.00" → 1200.00).
2. Convert all dates to YYYY-MM-DD.
3. Fix only obvious OCR typos in strings (e.g. "S1200" on a total line → 1200).
4. If a value is missing or unreadable, set null. Do not guess.

### Output Constraint
Output ONLY valid, parseable JSON inside a ```json code block.
No conversational preamble or postscript.

### Input Text:
[PASTE YOUR MESSY PDF TEXT HERE]

Adapt the skeleton to your domain (requirements, test cases, BOM lines, CI job metadata)—the pattern is schema + normalize + null + zero-chatter.

---

Template B — Messy table text → clean CSV

You are an expert tabular data processor. Extract tabular information from the raw text below as clean CSV.

### CSV Specification
Headers (in order): Date, Transaction_ID, Description, Category, Amount
Delimiter: comma

### Parsing Rules
1. Columns delimited by commas.
2. If a field contains a comma or double quote, wrap the field in double quotes
   and escape internal quotes per RFC-style CSV (e.g. "Services rendered, item 1").
3. Amount is a clean numeric float (e.g. -45.50 or 1500.00). No currency symbols.
4. Missing values → empty cell (nothing between commas). No "N/A".
5. Ignore PDF boilerplate (page numbers, "Confidential", watermarks).

### Output Constraint
Output ONLY the CSV inside a ```csv code block. No explanations.

### Input Text:
[PASTE YOUR MESSY PDF TEXT HERE]

---

Advanced: two-pass extraction (clean, then structure)

If text is massive or exceptionally garbled, do not ask one prompt to both repair layout and invent schema-perfect rows.

1. Pass 1 — Cleanup only
   Reconstruct broken text into clean chronological paragraphs or Markdown tables without changing meaning (no new facts).
2. Pass 2 — Extraction
   Feed the cleaned text into Template A or B.

Separating cleaning from structuring reduces error rate the same way Part II separates quote from interpretation.

Validate outside the model: jsonschema, CSV parsers, type checks, row counts, checksum totals. The model’s “valid-looking” fence is not a contract until your code accepts it.

---

Part IV — Massive PDFs: parse layout, then choose architecture

Thousand-page filings, manuals, and archives hit two bottlenecks:

1. Formatting corruption on extract — scrambled tables, multi-column mash, lost reading order.
2. Context degradation — lost-in-the-middle (Part I), hallucination under load, token/budget limits.

Cure is rarely “bigger paste into chat.” It is layout-aware parse → then an architecture matched to the question.

Step 1 — Layout-aware parsing (before any LLM “importance”)

Standard flatteners (PyPDF2-style text dump, naive OCR) destroy structure. AI-native / layout-aware tools rebuild hierarchy (headings, tables, figures) into Markdown or JSON before importance proxies run—so Proxy 3 (structure) has something real to latch onto.

Rule of thumb: if tables matter, do not feed raw pdftotext into Part III templates and hope. Parse to Markdown/JSON first; then schema-extract.

Tools evolve; verify current APIs and licenses before production commitment. The architecture lesson is stable: structure in → structure out.

Step 2 — Architectures once you have clean Markdown/text

Pick by query goal, not by hype:

1. Contextual retrieval (precise facts)

Problem: Fixed 500-token chunks drop document-level anchors (“which company? which year?”) mentioned 50 pages earlier.

Technique: Before embedding, pass each chunk + a short document macro-summary through a lightweight model; prepend 50–100 tokens of context to the chunk, then embed.

Result: Fewer “orphaned number” retrievals in huge manuals.

Link: Fixes Part I Proxy 6c—retrieval defines the candidate set for “importance.”

2. GraphRAG (global / cross-chapter questions)

Problem: “What are the overarching themes?” has no home in a single chunk.

Technique: Extract entities and relations into a knowledge graph (e.g. Microsoft GraphRAG, LightRAG, or similar). Query walks nodes across pages.

Result: Multi-hop and holistic questions become graph traversal + generation, not one lucky vector hit.

3. Hierarchical chunking / RAPTOR-style trees

Problem: Flat chunks miss thematic nested structure.

Technique:

1. Leaves — paragraph chunks
2. Branches — summarize/clusters of leaves
3. Trunk — chapter-level summaries

Retrieve top-down: chapter → section → leaf.

Result: Part II “sequential section walk,” automated.

4. Map-reduce / refine (whole-document digests)

Map: Run the same extraction or section summary prompt per chapter/section independently.
Reduce: Consolidate section outputs into one master digest, risk register, or takeaway set.

Best for: Executive digests and audits that must cover the book without one giant context.

Link: Map step = Part II sequential processing; reduce step = your rubric, not the author’s intro.

5. Long-context models + prompt caching (direct Q&A)

For documents that fit a large window (order-of-magnitude: hundreds of thousands to ~1M+ tokens depending on product—measure your corpus), load cleaned Markdown and use prompt caching so the document is paid/indexed once and requeries are cheap/fast.

Best for: Interactive Q&A on one huge PDF you will interrogate many times.
Not a free lunch: still lost-in-the-middle risk—combine with Part II brackets and needles for critical spans.

---

Recommendation matrix (PDF scale)

---

Full stack picture (Parts I–IV)

  PDF bytes
      │
      ▼
  layout-aware parse  →  Markdown / JSON  (Part IV Step 1)
      │
      ├── small / scoped text ──► Part II query prompts (claims, risks)
      │                              or Part III schema (JSON/CSV)
      │
      └── huge corpus ──► architecture (Part IV Step 2)
              ├── contextual retrieval
              ├── GraphRAG
              ├── hierarchical / RAPTOR
              ├── map-reduce
              └── long-context + cache
      │
      ▼
  “importance”  =  your predicates + schema + retrieval
                   NOT the model’s summary genre

  Soft: “most important”
        → proxies (Part I)
  Hard: “Extract WHERE … / schema … / pages …”
        → query engine (Part II)
  Rows: JSON/CSV rules
        → parsers, not storytellers (Part III)
  Scale: parse layout + pick architecture
        → systems, not one mega-prompt (Part IV)

---

Closing

Part I: Importance-in-the-model is proxy math, not moral judgment.
Part II: Importance-in-the-workflow is a query plan—ban generic summarize, quote-first, adversarial roles, deliberate scope.
Part III: Importance-for-pipelines is a schema—normalize, null policy, zero-chatter, few-shot, two-pass clean→structure.
Part IV: Importance-at-document-scale is systems design—layout-aware parse first; then retrieval, graphs, trees, map-reduce, or long-context cache by goal.

An LLM will not become a wise colleague who “just knows what matters.” It can become a controlled extract and transform stage in a pipeline you own—still fallible, always validated outside the chat UI—while human stakes stay in the rubric you refuse to outsource.

---

Open edge: If “importance” is mostly proxied statistics, what happens when the source was written by another model, your extraction prompts were model-written, and your chunk summaries for contextual retrieval were model-written too? How do you detect a hall of mirrors before a JSON row or a GraphRAG edge ships into a real decision?