The 5 concepts of D5.6

Tap any concept to jump to its 5-card cluster.

• 1Why This Module ExistsThe D5.5/D5.6 gap and how to close it
• 2Information ProvenanceClaim → source mappings in the output schema
• 3Temporal As-Of ReasoningPoint-in-time correctness with valid_from / valid_to
• 4Stratified + Field ConfidenceSampling buckets and per-field uncertainty
• 5Synthesis OutputEstablished vs. contested vs. single-source claims

The D5.6 gap

An audit of this course against the official Claude Certified Architect — Foundations blueprint surfaced that Domains 5.5 (human review & confidence calibration) and 5.6 (provenance & uncertainty) were the most under-covered topics. Cert tips were inserted into M09, M11, M17, and M18 to flag them in their natural homes — but the cert tests this material as one discipline, not as scattered footnotes.

This module exists to close that gap. The five sub-topics share a single underlying skill: knowing what you don't know, and surfacing that uncertainty to humans in a structured form. Once you see them as one discipline, the cert questions stop looking like trick questions and start looking obvious.

The lab notebook discipline

BEFORE: A talented junior scientist runs an experiment and writes a one-line conclusion: "Compound X cures the disease." Confident, declarative, clean.

PAIN: Six months later the paper is challenged. Which run? Which sample size? Which buffer batch? When was it valid — before or after the equipment recall? The conclusion is unauditable. The science is unrecoverable. The reviewer cannot tell whether the result is correct, contested, or quietly retracted.

MAPPING: A lab notebook with provenance (which reagent), temporal metadata (which day), sampling notes (which subset of trials), and explicit disagreement flags ("two runs conflicted") is what a senior scientist produces. Domains 5.5 & 5.6 are the cert asking: does your agent produce a lab notebook, or a one-line answer?

Five sub-topics, one discipline

1. ProvenanceEvery claim points back to its source(s). Without this, you can't audit and can't recover from a retraction.
2. Temporal handlingEvery fact has a valid_from / valid_to. Without this, agents confidently report stale facts as current.
3. Stratified samplingReviewers see N items from each confidence bucket, not the top-N. Without this, defects in low-confidence work are invisible.
4. Field-level confidenceUncertainty lives per field, not per document. Without this, a 30% confidence field hides inside an 88% confident doc.
5. Structured synthesisOutput schema flags claims as established, contested, or single_source. Without this, disagreement is silently lost.

These aren't five techniques. They're five expressions of one principle: make uncertainty legible.

When to apply D5/6 patterns

# Question: which D5/6 pattern does this scenario need?

IF the agent emits a factual claim:
  REQUIRE provenance # claim -> source_id

IF the fact can change over time
  (CEO, price, policy, coverage):
  REQUIRE temporal metadata
  # valid_from, valid_to, default = current

IF humans review a sample of outputs:
  REQUIRE stratified sampling
  # N from each confidence bucket

IF the output has multiple fields with
  independent risk profiles:
  REQUIRE field-level confidence
  # aggregate confidence hides defects

IF multiple sources contribute to a claim:
  REQUIRE synthesis status
  # established | contested | single

If the scenario triggers any of these, the cert-correct answer involves the matching pattern — not "make the model more accurate."

Misconceptions

Every claim, a pointer back

Information provenance is the property of an output schema in which every emitted claim carries a structured pointer back to its source(s). It is distinct from citation — citations are a writing convention; provenance lives in the schema.

The cert tests whether the output makes the claim → source link enforceable, not whether the answer happens to be right. {claim, sources:[id], confidence} is compliant; prose with parenthetical "[1][2]" markers is not, regardless of accuracy.

Wikipedia with vs. without citations

BEFORE: Picture a Wikipedia article with no citations. The prose flows beautifully, every claim sounds right, and a journalist quotes it confidently in a major piece.

PAIN: Six months later, the article's anonymous author admits paragraph 4 was fabricated. The journalist's editor asks: "Which paragraphs of your article relied on that source?" The journalist has no answer — the unsourced article gave them no thread to pull.

MAPPING: Wikipedia with [1][2] markers, but better — encoded in the schema, not the prose. With {claim, source_id, confidence} on every emission, a single retracted source ID instantly surfaces every affected downstream claim. No re-synthesis required.

From sources to auditable output

1. Assign stable source IDsEvery chunk, document, or DB row gets a deterministic ID before the agent ever sees it. The ID is what the schema will reference.
2. Extract claim-by-claim, not summaryAsk the model for distinct claims plus their supporting chunk IDs. A single "summary" collapses multiple claims and breaks provenance.
3. Emit structured output, not proseThe synthesizer returns {claims:[{text, sources:[id], confidence}]} — never a paragraph with brackets.
4. Maintain a reverse indexFor every source_id, store the set of claims that reference it. This is what makes retraction-handling fast.
5. On retraction, pull the threadLook up source_id in the reverse index, surface every dependent claim, mark them needs_review. No re-synthesis required.

Claim-source struct + lookup

# 1. The schema
STRUCT Claim:
  text: string
  sources: list[source_id]
  confidence: float       # 0.0 - 1.0
  status: "established" | "contested" | "single"

# 2. Build reverse index when synthesis runs
FUNCTION build_reverse_index(claims):
  index = {}
  FOR c IN claims:
    FOR sid IN c.sources:
      index[sid].add(c)
  RETURN index

# 3. On retraction, surface affected claims
FUNCTION on_retract(sid):
  affected = reverse_index[sid]
  FOR c IN affected:
    c.needs_review = true
  RETURN affected   # no re-synthesis pass

The schema does the work. The code is short because the structure carries the meaning.

Misconceptions

Facts have lifespans

"CEO is Alice" was true once and may now be false. Memory layers without temporal metadata cause agents to confidently report stale facts as current — one of the most-tested Domain 5.6 scenarios.

Every fact deserves {value, valid_from, valid_to, source}. The default fact query is "what's true now?" — WHERE valid_to IS NULL. The "as-of" query is "what was true at time T?" — WHERE valid_from <= T AND (valid_to IS NULL OR valid_to > T). Most temporal bugs come from omitting the valid_to predicate, which silently returns every historical version.

Forecast vs. record

BEFORE: A planning agent reads "expected high tomorrow: 78°F" from yesterday's weather feed and recommends a t-shirt to the user this morning.

PAIN: By the time the user reads the recommendation, "tomorrow" has come and gone. The actual high was 52°F. The agent treated a forecast (transient) as a record (permanent), and the user wears a t-shirt in the cold.

MAPPING: Every fact in your knowledge base has a valid_from and a valid_to. Forecasts have a near-future valid_to. Records have a permanent one. CEOs, prices, and policy versions have lifespans that need explicit tracking. Treating them as equivalent is the bug.

Two queries, one schema

1. Store every fact with boundsEvery row gets valid_from (when the fact became true) and valid_to (when it stopped, or NULL for "still true").
2. Append, never overwriteWhen a fact changes, set the old row's valid_to to now and insert a new row with valid_to = NULL. History is preserved.
3. Default query = "now"For "what's currently true?" always filter WHERE valid_to IS NULL. Forgetting this filter is the #1 temporal bug.
4. As-of query = explicit timestampFor "what was true at T?" filter valid_from <= T AND (valid_to IS NULL OR valid_to > T).
5. Pass the as-of date through the agentIf a user asks about 2023, the agent's tool call must include as_of: 2023-12-31, not just rely on default-now.

As-of query pattern

# Store every fact with bounds
STRUCT Fact:
  entity: string          # "Acme"
  attribute: string       # "ceo"
  value: string           # "Alice"
  valid_from: timestamp
  valid_to: timestamp | NULL   # NULL = current
  source: source_id

# Currently true (default query)
FUNCTION current(entity, attr):
  RETURN SELECT value FROM facts
    WHERE entity=entity AND attribute=attr
    AND valid_to IS NULL

# True at point in time T
FUNCTION as_of(entity, attr, T):
  RETURN SELECT value FROM facts
    WHERE entity=entity AND attribute=attr
    AND valid_from <= T
    AND (valid_to IS NULL OR valid_to > T)

The valid_to IS NULL filter is the load-bearing line. Forget it and you get every historical row.

Misconceptions

Sample the distribution, score the field

When you route extractions to human reviewers, sampling matters as much as the extraction itself. Aggregate confidence is too coarse. Top-N-by-confidence over-reviews the easy cases. Uniform sampling under-reviews the hard ones. The cert pattern is stratified sampling: N samples from each confidence bucket, so reviewers see the full distribution.

Field-level confidence is the second half. A doc with 88% overall confidence might have one field at 30% — the field that determines whether you approve a $50K healthcare claim. Aggregate hides per-field defects. Field-level scores let you escalate just that field, not the whole document.

Factory QC, by grade

BEFORE: A factory inspects only the shiniest widgets coming off the line — the ones that already look perfect. It feels rigorous; the inspector sees lots of widgets.

PAIN: They never find defects. The defective batches never reached the inspector's bench because the shiniest widgets were sorted out first. Same problem with top-N-by-confidence review of LLM extractions: reviewers see only the model's most confident work, learn nothing, and the bad extractions ship.

MAPPING: Stratified QC inspects N units from each grade tier — A, B, C. Defects across grades surface evenly. For LLM extractions: sample N from each confidence bucket (high, medium, low). Reviewers see where defects actually live (low confidence) in proportion to the rest of the distribution. Add field-level scores so an 88% doc with a 30% critical field doesn't slip through.

Bucket, sample, escalate

1. Score per field, not per documentEvery extracted field carries its own confidence. Aggregate doc-level confidence is a derived view, never the basis for review decisions.
2. Bucket by confidence bandSort fields into HIGH (>90%), MED (70-90%), and LOW (<70%) buckets. Bucket boundaries are policy decisions, but three buckets is the cert default.
3. Sample N from each bucketIf your reviewer budget is 6 items, take 2 from HIGH, 2 from MED, 2 from LOW — not 6 from HIGH.
4. Escalate critical low-conf fields directlyAny field below a hard floor (e.g., 40%) that drives a high-stakes decision bypasses sampling entirely and goes straight to a human.
5. Feed reviews back into calibrationReviewer corrections update the confidence model. Without the feedback loop, you sample forever without learning.

Stratified sample + field confidence

# Each extracted field carries its own score
STRUCT Field:
  name: string
  value: any
  confidence: float
  is_critical: bool

# Bucket fields, then sample from each
FUNCTION stratified_sample(fields, budget=6):
  high = [f FOR f IN fields IF f.confidence > 0.9]
  med  = [f FOR f IN fields IF 0.7 <= f.confidence <= 0.9]
  low  = [f FOR f IN fields IF f.confidence < 0.7]
  per_bucket = budget / 3
  RETURN random_pick(high, per_bucket)
       + random_pick(med,  per_bucket)
       + random_pick(low,  per_bucket)

# Hard escalation for critical low-conf fields
FUNCTION route(field):
  IF field.is_critical AND field.confidence < 0.4:
    RETURN "escalate_human"   # bypass sample
  ELSE:
    RETURN "sample_pool"

Misconceptions

Established, contested, or single

When sources agree, the agent says so confidently. When sources disagree, the agent must surface the disagreement — not silently pick one. This is the cert pattern that fails most often in practice: agents that "helpfully" reconcile contradictions by picking the more recent or more authoritative source and reporting it as fact.

The output schema must support a status: "established" | "contested" | "single_source" field. When status is contested, the output includes both source pointers in opposition — it doesn't pick a winner.

Meta-analysis paper

BEFORE: A meta-analysis paper reviews 10 studies on a drug's effectiveness. The author writes one clean sentence: "The drug is effective." Concise, decisive, citation-free.

PAIN: The clinical reviewer reading the paper has no idea whether "effective" means "7 studies all replicated the result" or "3 studies say yes, 2 say no, and the author picked the 3." Same sentence, vastly different evidence weight. The reviewer's downstream decision is now under-informed.

MAPPING: "Result X is replicated across 7 studies" reads very differently from "Result Y is supported by 3 studies and contradicted by 2." A reader can act on the first; the second requires more digging. Your synthesis schema must preserve that difference: established vs. contested vs. single_source, with all source IDs attached.

The synthesis pipeline

1. Per-chunk claim extractionEach source chunk goes to Claude with a structured tool that extracts every distinct claim plus per-claim confidence. Don't ask for a summary — that collapses claims.
2. Cross-chunk claim alignmentGroup claims that talk about the same fact. Two chunks both saying "CPT 99213 is Level 3" become candidates for an established merge.
3. Status assignmentMultiple sources agree → established. Multiple sources disagree → contested (both retained). Only one source → single_source (flag for review).
4. Emit the synthesis objectOutput a list of claims, each with text, sources, confidence, and status. The downstream consumer — human or agent — reads the status to act.
5. Never silently resolve a contradictionEven if the agent has a strong opinion about which source is right, the schema must surface the disagreement. Resolution is a downstream decision.

ProvenancedSynthesizer template

# Class: ProvenancedSynthesizer
CLASS ProvenancedSynthesizer:

  FUNCTION synthesize(chunks, query):
    claims = []
    FOR chunk IN chunks:
      extracted = extract_claims(chunk)   # tool call
      FOR c IN extracted:
        c.sources = [chunk.id]
        claims.append(c)

    # Align claims about the same fact
    groups = group_by_fact(claims)

    output = []
    FOR g IN groups:
      IF len(g) == 1:
        status = "single_source"
      ELIF all_agree(g):
        status = "established"
      ELSE:
        status = "contested"   # retain BOTH
      output.append({
        text: merged_text(g),
        sources: union_of_source_ids(g),
        confidence: aggregate_confidence(g),
        status: status
      })
    RETURN output

The else branch is the load-bearing one. It refuses to pick a winner.

Misconceptions

One question per concept

Tap a card to reveal the answer.