Spec: F86 — Extraction Faithfulness Contract

Status: in progress (backend shipped 2026-06-24; Contract Graph migration 2026-07-14)

The invariant was historically piecemeal — F08, F70, and F12 each carried a piece of the guard. The backend implementation and this Contract Graph contract now make it one measurable, enforced cross-surface product invariant. This spec is the single contract that makes faithfulness a measurable, enforced, cross-surface product invariant: zero invented payments, deadlines, or owners. It governs every AI surface that produces a structured item (F08 extraction, F70 chaos import, F83 pulse v2, F12 brief synthesis, F14 chat). It introduces no parallel provenance system — it consolidates the guards onto the existing spine (EvidenceSource + confidence policy + Rule 2).

1. Problem & CEO Job

Trust the signal. The pitch makes one hard, non-negotiable accuracy promise — the EVIDENCE slide’s scaling gate reads “ACCURACY: Zero invented payments, deadlines, or owners” — and the whole competitive thesis rests on it: “Every item cites its source line”, “Source traceability — every item cites the chat, screenshot, or voice note it came from”, “the difference between an answer and a record” (COMPETITION slide). A generic copilot can invent a plausible deadline because its output is a disposable answer. Lumi’s output is a record the owner acts on — calls a customer, chases a ₫50M deposit, sign-offs a build. If Lumi invents a payment, a deadline, or an owner once, the owner stops trusting the brief, and a brief the owner does not trust is worthless — they go back to scrolling four Zalo groups by hand. The asymmetry is the product: a missed signal costs little (tomorrow’s pulse repeats it); a fabricated signal costs the trust the product never gets back. Faithfulness is therefore not a feature on top of extraction — it is the product. Without a single enforced contract, faithfulness is whatever each surface’s author happened to remember to guard, which is exactly the state today: real but piecemeal, and unmeasured.

2. User Stories & Acceptance Criteria

• As an owner (CEO), I want every item in my brief to be traceable to an exact line I or my staff wrote, so that I can verify before I act and never get burned by an invented fact.
  • Given any AI-produced structured item (Commitment / Risk / Blocker / DecisionRequest / Task / brief line), when it renders, then it links ≥1 EvidenceSource whose recorded span is a verbatim substring of the cited source text (chat line / OCR text / transcript / spreadsheet row), normalized for whitespace and case only.
  • Given an item the AI proposed but for which no grounded span can be found in the source, when extraction finishes, then the item is dropped, not shown (fail-closed) and a structured warning is logged — never rendered as an asserted fact.
• As an owner, I want the numbers, dates, and names in an item to be the ones I actually wrote, so that Lumi never quietly substitutes a value.
  • Given an item carrying a money amount (e.g. ₫50M), a date/deadline, or an owner/assignee name, when it is produced, then each of those values must appear in — or be directly resolvable from — the cited span; an item asserting a value absent from its span is dropped (fail-closed), not silently corrected.
• As an operator, I want honest confidence everywhere, so that “1.0 = certain” never appears on an extracted fact.
  • Given keyword extraction, then confidence_score ∈ [0.60, 0.80]; given LLM extraction, then ∈ [0.85, 0.90]; 1.0 never occurs and confidence is never hardcoded (Rule 2; test-enforced).
• As the team, we want one shared guard, so that no surface re-invents (and weakens) faithfulness.
  • Given any AI surface that produces structured items, when it commits them, then it runs the same shared faithfulness validator — not a per-surface copy. A new surface that produces items without calling it is a release-blocking defect.
• As an owner whose message was too thin to ground an item, I want an honest empty/uncertain state, so that Lumi never fills the screen with a guess.
  • Given input from which nothing can be grounded, when extraction runs, then the surface shows a guided “we couldn’t find X in your message” / low-signal state — never an invented item.

Each criterion maps to at least one test in §7.

3. Scope Boundaries

• In:
  • The cross-surface faithfulness invariant (§2) and the single shared guard that enforces it.
  • Application to every structured-item producer: F08 keyword extraction, F70 chaos import (LLM + keyword), F83 pulse v2 industry/role extraction, F12 brief synthesis (summary may rephrase but may not introduce facts absent from the items), F14 chat (answers cite items, never invent).
  • The G5 accuracy benchmark (§5b): a fixed corpus + harness measuring invented-fact rate and recall/precision, run separately from unit tests.
  • Confidence-band enforcement (consolidating, not changing, the existing policy).
• Out:
  • The selection/ranking logic of each surface (which item to show) — F86 governs faithfulness, not prioritization. F12 §4b scale selection stays in F12.
  • The language of the output — that is the F37 contract; F86 governs whether the fact is real, F37 governs which language it is stated in. The two compose (see §10).
  • Production sampled-audit instrumentation of the invented-fact rate — that is F85 (this spec defines the offline benchmark; F85 measures the same invariant live via sampled audits + F15 feedback).
  • A new provenance store. F86 reuses EvidenceSource and adds only the small fields in §4 — no parallel system.

4. Data Model

Reuses the spine — no new core model. Every AI-produced item already links EvidenceSource (F07 pulses/models.py::EvidenceSource, UUIDModel, organization FK, source_type, title, content_text, metadata_json). The contract is enforced against this model, not by adding to it.

The historical gap where the span actually quoted was stored inconsistently is now closed by the append-only migration 0005_evidencesource_evidence_span_and_more. F70’s LLM path, F70 keyword path, and F08/F83 extraction paths persist the exact span on the shared evidence row; the guard stamps grounding_verified=True only after checking it.

Shipped (small, additive) — EvidenceSource.evidence_span:

• evidence_span = models.TextField(blank=True) on the existing EvidenceSource model (still UUIDModel + organization FK; tenant-scoped exactly as today). It holds the verbatim substring of content_text that grounds the linked item. blank=True so legacy rows and rule-based items (deal brief items, which carry related_deal provenance instead of a span — see F12 §4) are valid.
• Shipped — grounding_verified = models.BooleanField(default=False) stamped True by the shared validator when it has confirmed evidence_span is a verbatim substring. This makes “did the guard run on this item?” queryable for the F85 production audit, without a second table.

Migration is append-only; no applied migration was edited. Both fields remain optional, so this is a non-breaking addition. State-machine diagrams are unaffected (no status field added). The accepted placement on EvidenceSource is recorded in ADR-027.

5. API Contract

F86 is not a new endpoint. It is a guarantee enforced inside the existing services, plus an offline benchmark target. No request shape changes; no new route; tenant scoping is unchanged (every EvidenceSource and item is already scoped by organization_id, and the validator never widens scope — it only reads the source text already loaded in-process).

5a — The shared faithfulness guard (single source of truth)

A single reusable function in backend/apps/ai/faithfulness.py, called by every structured-item producer before commit. It consolidates the guards that already exist in the repo:

def assert_grounded(
    *,
    span: str,
    source_text: str,
    confidence: float,
    mode: str,            # "keyword" | "llm"
    facts: dict | None = None,   # {"amounts": [...], "dates": [...], "owners": [...]}
) -> GroundingResult:
    """
    Returns GroundingResult(ok=bool, clamped_confidence=float, reason=str|None).

    Fail-closed contract:
      (a) span must be a verbatim substring of source_text after the SAME
          whitespace/case normalization the surfaces already apply
          (collapse runs of whitespace, casefold). If not → ok=False.
      (b) every value in `facts` (money amounts, dates/deadlines, owner/assignee
          names) must appear in — or be directly resolvable from — span. A value
          absent from span → ok=False (no invented payments/deadlines/owners).
      (c) confidence is clamped to the honest band for `mode`
          (keyword [0.60,0.80], llm [0.85,0.90]); never 1.0, never hardcoded.
    On ok=False the caller DROPS the item and logs a structured warning
    (see §9) — it never renders it.
    """

Real guards this consolidates (verbatim from the repo today — F86 does not invent them, it unifies them):

• F70 LLM path (apps/chaos/services.py) — the prompt already demands: “For every item, copy a verbatim evidence_span — an exact substring of the source that proves the item. If you cannot quote it, do not emit it.” and the post-filter already enforces it:

  # Hallucination guard: the quote must exist verbatim in the source.
  if span not in content:
      logger.info("chaos_llm_dropped_unsourced_item type=%s", item_type)
      continue

  with confidence clamped by _clamp_llm_confidence → max(0.85, min(0.90, v)).
• F08 keyword path (apps/ai/services.py) — _answer_title(answer_text, patterns) sets each item’s title to the matched sentence from the source, never a raw_text[:100] dump; confidence is slot-based 0.6/0.75, never 1.0 (_compute_confidence).
• F12 / F37 brief synthesis (apps/ai/prompts.py brief surface) — the OPERATING RULES already bind: “Use ONLY the facts in the items. Never invent owners, numbers, dates, or items. Do not add items not in the list…”; provider failure falls back to deterministic templates (the brief never invents to fill space).
• F14 chat (apps/ai/prompts.py chat surface) — “Answer ONLY from the DATA section… never invent names, numbers, dates, owners, or items… A fluent but unsupported answer is the worst possible failure.”

F86’s job is to make these one function, called everywhere, with the facts cross-check (b) added — which no surface enforces today (they check the span, not that the amount/date/owner inside the item came from that span).

5b — Enforcement points per surface

Each surface calls assert_grounded(...) (5a) at its enforcement point; an item that fails is dropped + logged, never shown. AI output rule (Rule 2): every surviving item links an EvidenceSource with a populated evidence_span; confidence honest per band.

6. UI / UX

F86 has no own UI — its UX is the absence of invented facts on every AI surface, and the honest empty/uncertain states the existing surfaces already specify:

• Dropped item → honest copy, never a guess. When the guard drops items and nothing groundable remains, the surface shows its existing low-signal/empty state, not a fabricated brief:
  • F70: chaosImport.error_low_signal / the guided “gửi mẫu chi tiết hơn” state (already specified in F70 §9).
  • F08/F83: a no-signal pulse → zero items, pulse still processed (F08 §2), with the empty-section placeholder copy in the Brief (F12 §6: “Không có rủi ro đang mở 🎉”).
• Uncertain, not asserted. Items below the confidence floor render as uncertain suggestions (F70 §2), never as hard commitments.
• Evidence is always reachable. Every rendered item exposes its evidence_sources (F12 §5 nested shape: source_type, title, snippet) so the owner can tap to see the source line — the visible proof that the fact is a record, not an answer.
• No new i18n namespace; F86 reuses existing keys (chaosImport.*, brief.*). DESIGN.md / TASTE.md pre-flight applies to any surface that renders a “we couldn’t find X” state, but no new component is introduced.

7. Test Plan

Backend coverage stays ≥97%. The G5 harness (#10) is not part of the coverage-gated unit suite — it is a separate target (§5b) run on demand / in a scheduled job, because it needs the labelled corpus and is allowed to be slow.

5c — The G5 accuracy benchmark (measurable)

Aligned with field-tests.md §15 + gate G5: “100 samples benchmarked; zero severe hallucination pattern”; pass bar includes “0 repeated severe hallucination pattern” and “100% of accepted items have source evidence.”

• Corpus: a fixed set of N ≥ 100 real Vietnamese field-service inputs (mix per field-tests §15: 30 Zalo/WhatsApp chat · 20 Excel/project list · 15 customer · 15 supplier · 10 staff/site · 5 voice-note transcript · 5 OCR text), each with human-labelled expected items (project, task/commitment, assignee, deadline, external party, risk, decision, suggested next message, confidence, evidence quote). Lives in backend/tests/fixtures/g5_extraction/ (corpus + labels.json), referenced from docs/strategy/field-tests.md §15. Real samples are scrubbed of PII before commit; if any sample cannot be scrubbed it stays out of the repo and the corpus notes the gap.
• Targets (honest):
  • Invented-fact rate = 0 — zero invented payments, deadlines, or owners across the corpus. This is the hard gate; any non-zero count fails the benchmark (it is the pitch’s ACCURACY gate).
  • Recall ≥ 0.80 on explicit items (field-tests §15: 90%+ for explicit assignee + commitment + deadline is the aspiration; the benchmark records the real number and does not round up).
  • Precision ≥ 0.85 — of items produced, the share that are correct + grounded. 100% of accepted items carry source evidence (this is structurally guaranteed by the §5a guard, and the benchmark asserts it).
• How it runs: make benchmark-extraction → a pytest module test_g5_extraction.py marked @pytest.mark.benchmark and excluded from the default -m "not benchmark" unit run. It runs each corpus sample through the real extraction path (provider mocked or a recorded fixture so the benchmark is deterministic in CI; a separate live-provider mode is opt-in), diffs produced items against labels, and prints a per-sample table mirroring field-tests §15 plus the three headline numbers. The build fails if invented-fact rate ≠ 0.
• Relationship to production: the offline benchmark proves the gate before shipping; F85 proves it continuously in production via sampled audits + F15 owner-correction feedback (an owner correcting an assignee/date is a live invented-fact signal). The two share this spec’s definition of “invented fact.”

8. Current Behavior

The backend invariant is now unified and fail-closed:

• F70 (backend/apps/chaos/services.py) routes keyword and LLM proposals through the shared guard, persists evidence_span, stamps grounding_verified, and drops/logs unsupported items.
• F08/F83 (backend/apps/ai/services/extraction.py) use the same guard for pulse extraction, including the amount/date/owner cross-check and tenant-scoped evidence link.
• F12/F14 use summary_introduces_new_fact after model output; an invented amount falls back to the deterministic grounded answer/summary and logs the miss. This synthesis post-check is currently amount-scoped by design; owner/date extraction from arbitrary prose remains a follow-up.
• G5 (backend/tests/benchmark/test_g5_extraction.py) runs the deterministic benchmark and hard fails a non-zero invented-fact rate. The checked-in seed passes, but it is not yet the N>=100 real field corpus required by the pitch gate.

The Contract Graph binds these claims to executable backend proofs. F85 owns live sampled accuracy measurement and no F86 status implies pilot success, browser success, or named-device runtime proof.

9. Error States

Logging precedent: F37 already logs wrong_language_delivered to Sentry with the locale pair as the metric that decides whether a provider is fit; F86 mirrors this — faithfulness_dropped_* counters per surface are the metric that decides whether an extraction provider/model is fit, and feed the F85 production audit. Never silently swallow a drop without a log; never fabricate to fill a screen.

10. Diagrams & Docs Touched

• DIAGRAMS.md: AI pipeline — add the faithfulness guard stage between “extraction/synthesis” and “persist/deliver” (the fail-closed drop point), shown as shared across all AI surfaces. Sits alongside the F37 prompt-contract stage (faithfulness = is it real; F37 = what language).
• backend/DIAGRAMS.md: AI pipeline — the shared assert_grounded call on each producer’s path; the shipped EvidenceSource.evidence_span / grounding_verified fields.
• GLOSSARY.md: add Faithfulness (the contract: zero invented payments, deadlines, owners), Grounding (an item is grounded iff a verbatim span of the source proves it), Evidence span (the exact source substring that grounds an item).
• ADR-027: the fail-closed faithfulness decision is recorded in docs/ADR/027-fail-closed-faithfulness.md.
• Cross-spec references are now aligned: F08 §real-mode, F70 §5/§11, F83 extraction, F12 §4a, F14, F37 §10 (compose note), and F85 (shares this definition).

11. Implementation Audit (2026-06-24 — backend shipped)

Built and shipped (backend). The 5 previously-unbuilt rows are now implemented; the 4 pre-existing guards are consolidated onto the shared function without weakening any.

Summary: 9/9 implemented in the backend. Coverage: apps/ai/faithfulness.py 100%; full backend suite green (4022+ tests pass, total ≥97%). The fail-closed invariant is recorded in ADR-027. The G5 benchmark runs and passes on a seeded synthetic corpus; populating the full N≥100 real scrubbed corpus is the remaining honest gap (the harness and the invented-fact-rate==0 gate already work).

Known design choice (chaos LLM secondary attribution). Where the chaos LLM proposes an assignee the span does not support, that attribution is stripped (set to None, logged) rather than dropping the whole grounded item — Lumi never asserts an owner the span cannot prove, while the real item survives (consistent with ES2’s “never substitute”). The item’s owner-facing title is still fully facts-checked (an invented amount in the title drops the item).

Known scope (brief/chat synthesis post-check). summary_introduces_new_fact (F12 ES4 / F14) checks for an introduced money amount only — the highest-trust-risk fabricated fact, and the one with unambiguous markers (₫/VND/units) that make detection in free-form summary text reliable. Owner-name and date checks on synthesis output are a tracked follow-up: extracting them from arbitrary prose without NER is noisy and would trigger false fallbacks on legitimate rephrasings. Extraction-time items (F08/F70/F83) are fully facts-checked (amount + date + owner) via assert_grounded; only the synthesis post-check is amount-scoped today.

12. Open Questions / Decisions

• Real G5 corpus — the deterministic harness and zero-invention gate are shipped, but the N>=100 scrubbed field corpus still needs consent, annotation, and product-owner sign-off.
• Production audit — F85 must attach sampled accuracy events and F15 owner-correction feedback before the live accuracy gate can be called passed.
• “Directly resolvable” for owners/dates — how lenient is the facts cross-check? An owner name may be a first name in the span resolved to a full OrganizationMember; a relative date (“thứ 5”) is resolved to an absolute date. The cross-check must accept resolution from the span but reject invention beyond it. The exact normalization rules are tuned against the G5 corpus (§5c) and must be conservative — when in doubt, drop.
• Confidence floor for auto-create vs uncertain — shared with F70 §12; tune against G5.
• Benchmark provider mode — deterministic (mocked/recorded fixtures) in CI vs a periodic live-provider run that exercises the real model. Recommendation: both — deterministic gate in CI, scheduled live run feeding F85.

13. Bug / Change Log

2026-07-14 — Contract Graph migration and implementation-truth correction

Type: Contract Graph / Documentation Truth Fix

Summary:

• Added the canonical contracts/features/F86.cue contract for the shared validator, consuming surface post-checks, and deterministic G5 harness.
• Bound the contract to collected backend proof IDs for substring grounding, invented-fact rejection, honest confidence, fail-closed surface behavior, and seeded corpus shape.
• Corrected this spec’s stale pre-implementation wording: EvidenceSource.evidence_span and grounding_verified are shipped, ADR-027 is accepted, and the backend guard exists.
• Kept the honest gaps visible: the seed is not yet the N>=100 real field corpus, live F85 accuracy evidence is not proven here, and consuming-surface browser/native runtime belongs to those features.

2026-07-02 — Fix stale test path: test_brief_synthesis.py → test_faithfulness.py in §7 row 8 (batch 3)

Type: Documentation

Summary:

• §7 row 8 referenced backend/tests/test_brief_synthesis.py — this file does not exist (ls backend/tests/ confirms). The brief-synthesis faithfulness test lives in backend/tests/test_faithfulness.py (see TestBriefSynthesis::test_brief_synthesis_rejects_invented_amount), verified by grep.
• Updated the evidence path in row 8 to backend/tests/test_faithfulness.py.
• No code changes.

2026-07-02 — §11 Row 5 faithfulness function corrected for F12/F14 (batch 12)

Type: Documentation Truth Fix

Summary:

• §11 Row 5: Corrected the evidence claim that F12 synthesis and F14 chat call assert_grounded/check. Those surfaces call faithfulness.summary_introduces_new_fact() (verified: backend/apps/ai/services.py line 1766 for F12, line 1998 for F14). assert_grounded/check is called only by the chaos LLM+keyword and F08/F83 pulse extraction paths. Row 8 already documented this correctly; Row 5 now cross-references Row 8 for F12/F14 rather than naming the wrong function.

2026-07-02 — Spec drift corrected (batch 5)

Type: Docs

Summary:

• Status already correct (in progress (backend shipped 2026-06-24)); no change.
• All §2 ACs marked [x] per §11 (9/9 rows implemented, 100% faithfulness.py coverage):
  • AC1 (verbatim-span guard): confirmed by §11 row 1 (assert_grounded shared guard) + §7 row 1 (TestGroundingGuard, green).
  • AC2 (fail-closed drop): confirmed by §11 rows 1 + 5 + §7 row 5 (green) and existing test_chaos.py::test_no_item_without_evidence_span.
  • AC3 (amount/date/owner cross-check): confirmed by §11 row 6 (facts= argument) + §7 rows 2–4 (TestInventedFacts, green).
  • AC4 (confidence bands): confirmed by §11 rows 2–3 (_clamp, bands enforced) + §7 row 6 (TestConfidenceBands, green).
  • AC5 (shared guard on every surface): confirmed by §11 row 5 (F08/F70/F83/F12/F14 all call assert_grounded) + §7 row 7 (TestAllSurfacesGuarded, green).
  • AC6 (honest empty/low-signal state): confirmed by §11 row 8 (ES4 synthesis fallback to deterministic template) + §9 ES3 (drops → existing low-signal states, never invented items).

2026-06-24 — Implemented F86 (backend)

Type: Feature (implementation)

Summary:

• Shipped the single shared faithfulness guard apps/ai/faithfulness.py (assert_grounded + check + log_drop + summary_introduces_new_fact) and wired it into every structured-item producer: F70 chaos LLM (_llm_proposals) and keyword (_keyword_proposals), F08/F83 pulse extraction (extract_pulse_entry), F12 brief synthesis (_synthesize_brief, ES4 fallback), and F14 chat (answer_chat_message).
• Added the programmatic amount/date/owner cross-check (new — no surface enforced it before), including Vietnamese money normalization so ₫50M / 50tr / 50 triệu / 50.000.000 match by value.
• Added the authoritative EvidenceSource.evidence_span (verbatim grounding substring) and grounding_verified fields (migration pulses/0005_evidencesource_evidence_span_and_more), stamped by the chaos + pulse paths.
• Added the G5 benchmark: make benchmark-extraction → backend/tests/benchmark/test_g5_extraction.py (marked @pytest.mark.benchmark, excluded from the default unit gate via -m 'not benchmark'), with a seeded labelled corpus in backend/tests/fixtures/g5_extraction/. Headline: invented-fact rate 0.000, recall 0.909, precision 0.968.
• Wrote ADR-027 (fail-closed faithfulness), added GLOSSARY entries (Faithfulness, Grounding, Evidence span), and updated DIAGRAMS.md + backend/DIAGRAMS.md (the shared assert_grounded stage + the new EvidenceSource fields).

Reason:

• §11 rows 5–9 were unbuilt: no shared function, no facts cross-check, no evidence_span field, no brief post-check, no benchmark. F86 consolidates the piecemeal guards and makes faithfulness a measurable, enforced cross-surface invariant.

Impact:

• Backend behavior strengthened (no existing test weakened; chaos/pulse/synthesis tests green). Tests: backend/tests/test_faithfulness.py (42, faithfulness.py 100% covered), test_multitenancy.py::TestFaithfulnessCrossTenant (2), plus the G5 benchmark. Full suite: 4022+ pass, total coverage ≥97%.

Notes:

• Chaos LLM secondary assignee not supported by the span is stripped (not dropped) — see §11 “Known design choice.” No frontend change (F86 has no own UI).

2026-06-24 — Spec created

Type: Feature (spec)

Summary:

• Created F86 to make extraction faithfulness — zero invented payments, deadlines, or owners — a single, measurable, enforced cross-surface product invariant. Consolidates the piecemeal guards that exist today (F70’s span not in content drop + _clamp_llm_confidence; F08’s _answer_title
  • slot-based confidence; F12/F37’s “never invent owners, numbers, dates” prompt rules) into one shared assert_grounded guard called by every structured-item producer, adds a programmatic amount/date/owner cross-check (not enforced anywhere today), proposes an authoritative EvidenceSource.evidence_span field, and defines the G5 accuracy benchmark (≥100 labelled Vietnamese field-service samples; invented-fact rate must be 0) that verifies the pitch’s ACCURACY scaling gate.

Reason:

• The pitch’s EVIDENCE slide gate — “ACCURACY: Zero invented payments, deadlines, or owners” — and the COMPETITION framing — “the difference between an answer and a record” — make faithfulness the product, but no single spec made it an enforced, benchmarked invariant. Guards existed per surface, unmeasured.

Impact:

• New draft feature F86 in the registry. No code shipped by this spec. Touches the design of F08, F70, F83, F12, F14 (shared guard) and F85 (shared definition for production audits). Recommends an ADR for the fail-closed invariant.

Tests:

• Test plan defined (§7); G5 benchmark target (§5c) specified; none implemented yet.

Notes:

• Reuses the existing spine (EvidenceSource + confidence policy + Rule 2) — no parallel provenance system. The fail-closed drop must always be logged (mirroring F37’s wrong_language_delivered); a drop that fills the screen with a guess is the failure this spec exists to prevent.