Spec: F08 — AI Extraction

Status: shipped · Audited: 2026-07-14

1. Problem & CEO Job

Trust the signal. This is the plumbing that turns chaos into the graph: six free-text answers in, structured Commitments/Risks/DecisionRequests out. Extraction itself is table-stakes — any AI can parse text. The moat is what it builds: the accumulated multiplayer data graph (customers/staff/tasks/commitments/risks/projects/suppliers) that no Zalo thread or Excel file holds. The value is converting the owner’s messy Zalo/Excel/screenshot reality into that graph, one pulse at a time. The quality bar is asymmetric: a missed signal costs little (tomorrow’s pulse repeats it); a fabricated or inflated signal costs trust the product never gets back. Fewer, honest, traceable items beat coverage — always.

This extraction also powers the wedge: the F70 Chaos Import → Boss Brief flow runs ON this pipeline — pasted Zalo/Excel/screenshot is just another input source feeding the same structured-item output.

2. User Stories & Acceptance Criteria

• As a CEO, I want every extracted item traceable to who said what, so that I can verify before acting.
  • Given any AI-created item, then ≥1 EvidenceSource linked (commitments, risks, decisions) AND extracted_from_pulse_entry set (commitments ✅, risks ✅, decisions ✅ — FK exists at operations/models.py:154; set in LLM path and keyword Commitment/Risk paths; keyword Decision path does not set it) — a floating insight is a release-blocking bug.
• As a CEO, I want honest confidence, so that I learn when to trust Lumi.
  • Given keyword extraction, then confidence_score ∈ [0.60, 0.80], computed from signal strength; LLM mode 0.85–0.90; 1.0 never occurs (test-enforced).
• As a member, I want my words to become readable items, so that the Brief doesn’t echo my raw text back.
  • Given extraction, then item titles are the matched sentence/clause — never a raw_text[:100] dump (see §9).
• As an operator, I want extraction to be deterministic in mock mode, so that dev/CI behave identically (ADR-002).
• Given one pulse, then ≤10 items total (more = noise; confidence-filter before create).
• Given extraction failure, then PulseEntry.status=failed and the error is retrievable (error_message field set by both submit_pulse and /process/ — see F07 §13 ES2/ES7); partial item creation must not mark processed.

Extraction contract (mock mode)

3. Scope Boundaries

• In: pulse → items pipeline, LLM extraction (primary) + keyword fallback behind LLM_PROVIDER, evidence linking, confidence policy, status transitions.
• Out: extraction from chat or tasks, embeddings/semantic search, model fine-tuning, auto-actions (extraction proposes; humans/Brief dispose).

§3a — LLM Extraction Contract (primary path, ADR-031)

When LLM_PROVIDER != "mock", extraction calls the configured provider via _extract_via_llm() before the keyword loops. On success, keyword extraction is skipped entirely. On LLMProviderError or invalid JSON, a warning is logged and execution falls through to the keyword path — the Brief must always have items.

System prompt: build_system_prompt(submitter_member, "extraction") — English instructions + member-language response directive (F37). The _EXTRACTION_RULES block defines signal types (commitment/risk/decision), grounding rules, and verbatim evidence span requirement.

User prompt: build_extraction_prompt(qa_text, today_iso) — concatenated Q&A text + today’s date + the JSON output schema.

Output schema (returned by LLM as response_json=True):

{
  "items": [
    {
      "type": "commitment" | "risk" | "decision",
      "title": "concise summary ≤80 chars in the pulse language",
      "evidence_span": "verbatim substring from the input text",
      "confidence": 0.85–0.90,
      "due_date": "YYYY-MM-DD or null",
      "owner_name": "person name or null",
      "risk_type": "cash" | "customer" | "delivery" | "operations" | null,
      "severity": "low" | "medium" | "high" | "critical" | null
    }
  ]
}

Faithfulness check (F86): each item’s evidence_span must be a verbatim substring of qa_text; items failing this check are dropped (not the whole batch). Confidence is clamped to [0.85, 0.90] regardless of what the LLM returns.

Cap: ≤10 items total (same as keyword path; _cap_extracted_items applies after both paths).

Do NOT grow keyword lists for new signal vocabulary. Instead, update the _EXTRACTION_RULES block in prompts.py so the LLM understands the new signal. Keyword patterns (_RISK_PATTERNS, _COMMITMENT_PATTERNS, _DECISION_PATTERNS) remain as the fallback floor for mock mode and LLM failures — they should be touched only when mock-mode coverage genuinely regresses.

4. Data Model

No own models. Writes Commitment/Risk/DecisionRequest (F09–F11), reads PulseEntry, links EvidenceSource (F07). All writes carry organization_id from the pulse — extraction must never widen tenant scope.

5. API Contract

Service layer only: apps/ai/services/extraction.py::extract_pulse_entry(pulse_entry_id, evidence_id=None, organization_id=None). Invoked by POST /api/pulses/<id>/process/ (and, per F07 decision, by submit auto-trigger, which passes its own evidence_id). Receives IDs, not objects (Celery-safe). Production HTTP callers pass the resolved organization ID so the initial pulse lookup is tenant-scoped. Early-returns if status == "processed" (idempotent retry guard). The optional organization argument is a temporary compatibility seam for legacy internal callers and offline tests; F08 remains blocked until those callers are migrated and the argument becomes required.

6. UI / UX

No direct UI. Its UX is the quality of items appearing in /intel/commitments, /intel/risks, /intel/decisions, and the Brief — which is why title readability (§2) is an acceptance criterion here, not a nicety.

7. Test Plan

8. Current Behavior

Shipped with mock-default local development and Gemini, OpenRouter, and DeepSeek support. Extraction creates evidence-linked operational items with bounded confidence. Before any supported external-model submission, known email/phone/credential patterns are minimized and user/tool payloads are marked as untrusted data beneath the system boundary across complete, tool-call, and stream paths. This does not represent arbitrary business text as anonymous. Numeric invoice, order, customer-ID, and currency-amount facts remain intact rather than being mistaken for unformatted phone numbers.

F86 (2026-06-24): each extracted item now routes through the shared apps/ai/faithfulness.py::assert_grounded guard (the matched-sentence span must be a verbatim substring of the answer; confidence clamped into the keyword band; ungrounded items dropped + logged), and the linked EvidenceSource is stamped with evidence_span + grounding_verified. See F86.

9. Error States

ES1 — PulseEntry.DoesNotExist (invalid pulse ID)

extract_pulse_entry calls PulseEntry.objects.get(id=pulse_entry_id) without a DoesNotExist guard. If the pulse ID is invalid or the pulse was deleted, DoesNotExist propagates unhandled. The /process/ view calls self.get_object() first (which returns 404 if not found), so this path is unreachable via the API. Direct service-layer calls (e.g., from Celery tasks) would crash.

ES2 — Extraction failure on submit (caught)

submit_pulse wraps extract_pulse_entry in try/except. On exception: pulse.status = "failed" + logger.exception(...). The submission itself returns HTTP 201 — user answers are never lost. The failed pulse is visible on the frontend with a retry button. Covered: test_submit_extraction_failure_sets_failed.

ES3 — Extraction failure on /process/ retry (unhandled)

POST /api/pulses/<id>/process/ does NOT wrap extract_pulse_entry in try/except. If extraction fails on retry, the exception propagates as HTTP 500. The pulse status remains unchanged (not reset to failed). This is different from submit behavior. Covered: documented in F07 spec §11 ES6; red test test_red_r18.py::test_process_endpoint_sets_failed_on_error (xfail — unimplemented).

ES4 — Partial item creation before crash

If extraction creates a Commitment but crashes during Risk.objects.create, the pulse is NOT marked processed. The commitment persists in the DB. On retry (/process/), the early-return guard if pulse.status == "processed" does not trigger (status is still submitted or failed), so a new commitment may be created — resulting in duplicates. There is no transaction.atomic wrapping extraction. The break after first match limits blast radius to at most one duplicate per type.

ES5 — Missing evidence (evidence_id=None, no records)

If no EvidenceSource exists for the org+user, evidence is None. All if evidence: guards skip the .add(evidence) call. Items are created without any evidence link. This violates §2’s acceptance criterion (“every AI-created item: ≥1 EvidenceSource linked”). In practice, submit_pulse always creates the evidence before calling extraction, so this path only triggers for orphaned manual /process/ calls.

ES6 — LLM provider errors (real mode, future)

LLMProviderError from get_provider() (missing API key, unknown provider) or from provider.complete() (HTTP error, timeout, malformed payload) propagates unhandled. In mock mode (current default), extraction never calls the provider — keyword matching is fully local. Real-mode extraction (not yet built) will need try/except around provider calls with pulse.status = "failed" on error.

ES7 — Provider timeout (30s default)

GeminiProvider and OpenRouterProvider use timeout=30 on HTTP requests. A timeout raises requests.Timeout → wrapped in LLMProviderError. This is a real-mode concern only. In mock mode, extraction is in-process with no network call.

ES8 — Duplicate items on retry after partial failure

If extraction partially succeeds (e.g., commitment created, risk creation crashes) and the pulse is not marked processed, a retry via /process/ re-runs full extraction. The commitment loop matches the same answer again and creates a second Commitment. There is no deduplication check (e.g., “already has a commitment linked to this pulse”). The idempotency guard only triggers on status == "processed".

ES9 — /process/ endpoint evidence fallback

The /process/ view searches for evidence by (org, source_type=pulse_entry, created_by=pulse.user) ordered by -created_at. If multiple evidence records exist, it picks the most recent — which may not be the one from the original submission. This is a weaker contract than submit_pulse which passes the exact evidence.id.

ES10 — Race condition on concurrent submissions

Two simultaneous submit_pulse calls for the same (org, user, date) both hit update_or_create. One succeeds; the other either updates the same row (if it runs after the first commit) or raises IntegrityError (if both try to INSERT before either commits). Django’s update_or_create does not use SELECT ... FOR UPDATE, so the race window exists at the DB level.

ES11 — Exception during pulse.save() after extraction

After all items are created, extract_pulse_entry calls pulse.save() to set status="processed" and processed_at. If this save fails (DB connection lost, serialization error), the pulse stays in submitted status despite items existing. A retry would create duplicate items (same as ES8).

10. Diagrams & Docs Touched

DIAGRAMS.md AI pipeline · docs/AI_PIPELINE.md · docs/AGENTS.md → AI Agent.

11. Implementation Audit (2026-06-12)

Summary: 9/10 match; the service-layer tenant guard is partially closed. Production submit/process callers now pass organization_id, and the service rejects a mismatched organization before reading the pulse. The optional compatibility path remains a blocking gap until all internal callers are migrated. Gaps 7 and 8 fixed 2026-06-26; gap 9 closed 2026-07-02 (FK exists and set in primary LLM path).

Edge Cases Catalog (formerly §8, 2026-06-12; E1–E14): Key items integrated above. Remaining: E1 empty pulse → 0 items, processed (✅ test_no_extraction_on_clean_text); E3 whitespace-only answer → no match (correct behavior, no guard needed); E4 same keyword in multiple answers → break after first match, one item per type per pulse (covered by code, no dedicated test for second-match ignored); E5 multi-keyword single answer → up to 3 items (correct); E6 title truncation at 120 chars (no overflow risk); E7 slot-based confidence 0.6 vs 0.75 (✅ tested); E9 company detection post-creation — exception propagates, item already committed (best-effort enrichment); E10 resubmission update_or_create → orphaned items from previous extraction not cleaned up (documented, accepted); E11 draft status dead code; E13 extract_message_signal always 0.6 (correct, no slot context); E14 nested import inside loop (Python cache makes fast). Full catalog in git history.

12. Open Questions

• Real-mode prompt design (resolved — ADR-031): build_extraction_prompt() + _EXTRACTION_RULES in apps/ai/prompts.py define the contract. LLM is the primary path when LLM_PROVIDER != "mock"; keywords are the fallback. See §3a for the full contract.
• Vietnamese-only keyword classifiers (F69 blocker): Keyword fallback is Vietnamese-only. For orgs writing pulse text in English, Malay, Arabic, or Bahasa, none match — every item defaults to operations/internal. Deal context strings in ai/services.py also use Vietnamese violating F37 prompts-in-English rule. Resolve before F69 activates any non-Vietnamese org. Preferred fix: LLM-based classification (F37 already requires this).

13. Bug / Change Log

2026-07-14 — External-model privacy and instruction boundary

Type: Security hardening

• Added one shared payload boundary for known direct identifiers and credentials across Gemini, OpenRouter, and DeepSeek.
• Wrapped user and tool-result content as untrusted data for complete, tool, cache, and stream paths.
• Escaped untrusted text so customer/tool content cannot close reserved boundary tags.
• Tightened phone recognition to preserve ISO dates and ordinary hyphenated business references.
• Preserved explicitly labelled unformatted business IDs and currency amounts.
• Added payload regression tests and F08 Contract Graph proof; live-provider policy/eval evidence remains explicit.
• Extended the same boundary to OpenAI-compatible complete, tool-call, and stream payloads after final review found raw provider-specific paths.

2026-07-13 — Thread tenant context through production extraction callers

Type: Security hardening / Contract Graph proof

Summary:

• extract_pulse_entry accepts an explicit organization_id and applies it to the first PulseEntry lookup.
• The submit auto-trigger, /process/ retry endpoint, and demo seed path pass the organization context they already own.
• Added explicit same-tenant and cross-tenant service proofs. A mismatched organization raises PulseEntry.DoesNotExist before extraction or writes begin.
• The argument remains optional only for legacy direct callers and offline tests; F08.CUE records that migration as a blocking gap rather than claiming full closure.

2026-07-02 — Upgrade DecisionRequest FK gap to ✅ and fix confidence evidence (batch-2 spec audit)

Type: Documentation Fix

Summary:

• §11 row 9: upgraded from ⚠️ gap to ✅ fixed — extracted_from_pulse_entry FK exists on DecisionRequest (operations/models.py:154) and IS set in the LLM extraction path (services.py:651). The keyword fallback path (services.py:1022–1034) omits it, but that path is mock/fallback only under ADR-031. Summary gap count corrected from 2 to 1.
• §11 row 2: corrected evidence — there is no _compute_confidence function; confidence is assigned inline at ~922/~965/~1008 within each keyword extraction loop. Behaviour claim (slot-based 0.6/0.75) is correct.
• §2 AC row 1: updated the decisions FK note to reflect that the FK exists and is set in the primary path; keyword-path omission noted.

2026-07-02 — Stale test-class reference corrected (§7 row 5)

Type: Documentation Fix

Summary:

• §7 row 5 previously referenced tests/test_llm_extraction.py::TestLLMExtraction, a class name that never existed in the file. test_llm_extraction.py has always split coverage by item type. Updated to the three classes that actually assert LLM extraction of each item type: TestLLMExtractionCommitment, TestLLMExtractionRisk, TestLLMExtractionDecision (all verified by grep).

2026-06-26 — LLM-first extraction implemented (ADR-031)

Type: Feature (primary extraction path)

Summary:

_extract_via_llm() added to apps/ai/services/extraction.py:

• When LLM_PROVIDER != "mock", the LLM is called first via check_and_retry() (F37 language contract).
• User prompt: build_extraction_prompt(qa_text, today_iso) — verbatim Q&A text + JSON schema.
• System prompt: build_system_prompt(member, "extraction") — includes _EXTRACTION_RULES signal definitions and RESPONSE LANGUAGE directive.
• Response: JSON {"items": [...]} per F08 §3a schema; parsed and validated.
• F86 faithfulness guard: each item’s evidence_span must be a verbatim substring of the pulse text; items that fail faithfulness check are dropped (not logged as errors).
• Confidence clamped to [0.85, 0.90]; out-of-range values are silently clamped.
• Items with types already produced by F84 field-service ontology pass are skipped (avoids double-counting).
• On LLMProviderError or invalid JSON: logs warning, returns False, falls through to keyword extraction.
• On success (including 0 items): returns True, caps items, marks pulse processed — keyword loops skipped.

_EXTRACTION_RULES and build_extraction_prompt() added to apps/ai/prompts.py:

• _EXTRACTION_RULES defines signal types (commitment/risk/decision), grounding rules, confidence policy, and output language directive.
• build_extraction_prompt(qa_text, today_iso) → user turn with raw text + JSON output schema.
• Rule: do NOT add to _RISK_PATTERNS, _COMMITMENT_PATTERNS, _DECISION_PATTERNS for new vocabulary — update _EXTRACTION_RULES instead (ADR-031).

Tests: backend/tests/test_llm_extraction.py — 18 tests covering commitment/risk/decision creation, evidence linking, faithfulness guard, confidence clamping, LLM fallback to keywords, mock-mode bypass, zero-items case, multi-item, and prompt shape. All 18 pass.

2026-06-26 — Vocabulary expansion: one-pager coverage + transaction.atomic hardening

Type: Feature (extraction quality)

Summary:

Atomicity (§11 rows 7–8):

• extract_pulse_entry delegates to _extract_pulse_entry_atomic inside transaction.atomic() — all items + status=processed commit as one unit; no orphaned items on crash.
• POST /api/pulses/<id>/process/ now sets status=failed + error_message on extraction error instead of returning 500 with partial data.

Risk pattern expansion (_RISK_PATTERNS):

• Order cancellation: hủy, hủy đơn, khách hủy
• Operational cash dry-up: hết tiền, không có tiền, vốn hết
• Workforce unavailability: cách ly, bị cách ly, bị thương, nhập viện
• Financial penalty: bị phạt, phạt tiền, tiền phạt
• Supply in transit: chờ hàng, chờ vật tư, chờ linh kiện, chờ giao, chờ về
• Workload capacity: quá tải, không đủ nhân sự, thiếu người

Commitment pattern expansion (_COMMITMENT_PATTERNS): đảm bảo, nhận việc

Decision pattern expansion (_DECISION_PATTERNS): không có quyền, không được phép, không thẩm quyền, vượt thẩm quyền — captures authority-gap blockers that require CEO resolution.

Risk type classification (_infer_risk_type):

• Penalty keywords now route to cash (previously fell through to operations)
• Cancellation keywords now route to customer (previously fell through to operations)

Due-date parsing (_extract_commitment_due_date):

• thứ X tuần sau/tới → specific weekday of next week (e.g. “thứ hai tuần sau” → next Monday). Checked BEFORE plain “tuần sau” so the weekday is not dropped.
• trong tuần / tuần này (no explicit weekday) → coming Friday as default.

Tests: backend/tests/test_pulse_v2_p3.py — 12 new test classes covering each pattern group (196 total); TestOnePagerExactExample (9 cases) verifies the canonical one-pager input end-to-end.

2026-06-15 — Frontend route namespaced

Frontend route namespaced: /commitments -> /intel/commitments, /risks -> /intel/risks, /decisions -> /intel/decisions (app-wide IA restructure; API paths unchanged).

2026-06-14 — Strategy v2 realignment (positioning)

Type: Documentation Change

Summary:

• §1 reframed: extraction is no longer cast as “the transformation that justifies Lumi’s existence.” It is now framed as table-stakes plumbing; the moat is the multiplayer data graph (customers/staff/tasks/commitments/risks/projects/suppliers) the extraction builds by converting messy Zalo/Excel/screenshot input into structured items.
• Added a cross-link noting that the F70 wedge (Chaos Import → Boss Brief) runs ON this extraction pipeline — pasted Zalo/Excel/screenshot is an input source. The pipeline is not redesigned.

Reason:

• Aligns F08 with competitive-strategy-v2.md and the MVP_SCOPE.md “Positioning doctrine (strategy v2, 2026-06-14)”: AI is table-stakes; the data graph is the moat. No spec may claim AI “justifies Lumi’s existence.”

Impact:

• Positioning only — no behavior, API, data-model, test, or status change. The extraction pipeline, confidence policy, and evidence contract are untouched.

Tests: None — doc-only positioning edit.

Notes:

• No behavior change is implied or required by this realignment.

2026-06-14 — Restructure to TEMPLATE.md section order; expand §11 audit (SP fleet)

Type: Documentation Change

Summary:

• Numbered “Current Behavior” as §8.
• §8 Edge Cases absorbed into §11 note; §9 Error States stays §9; §10 Diagrams stays §10; §11 Implementation Audit (narrative §11 + table §11a) merged into single 10-row table; §12 Open Questions stays; §13 Bug Log stays.
• §11 audit: 10 rows, 6 ✅, 4 gaps (/process/ error handling, partial rollback, DecisionRequest FK, service-layer tenant guard).

2026-06-12 — Close extraction item cap acceptance criterion (A-r22)

Type: Documentation Change

Summary:

• Marked [x] for the acceptance criterion: “Given one pulse, then ≤10 items total.”
• Verified: _cap_extracted_items(pulse_id) exists in apps/ai/services/extraction.py, called at end of extract_pulse_entry. Tests in test_red_r18.py pass; xfail markers removed.

2026-06-12 — error_message field + Risk.extracted_from_pulse_entry FK shipped

Type: Feature

Summary:

• PulseEntry.error_message TextField added (migration 0002) — extraction failure detail is now retrievable.
• Risk.extracted_from_pulse_entry FK added (operations migration 0006) — mirrors Commitment’s existing FK for extraction provenance.
• services/extraction.py::extract_pulse_entry sets extracted_from_pulse_entry=pulse on created items.

2026-06-12 — Red tests for two unstruck acceptance criteria (E-r18)

Type: Test Coverage (RED)

Summary:

• Added tests/test_red_r18.py::TestF08ExtractionItemCap::test_extraction_caps_at_ten_items (xfail, strict).
• Added tests/test_red_r18.py::TestF08ExtractionFailure::test_pulse_entry_has_error_message_field (xfail, strict).
• Added tests/test_red_r18.py::TestF08ExtractionFailure::test_process_endpoint_sets_failed_on_error (xfail, strict).

2026-06-12 — Edge Cases and Error States sections added

Type: Documentation Change

Summary:

• Added Edge Cases (E1–E14): empty pulse, missing answers_json, whitespace answers, multi-match break, multi-keyword single answer, title truncation, slot-based confidence, evidence fallback ambiguity, company detection post-creation, resubmission update-or-create, draft dead code, cross-tenant service-layer gap, extract_message_signal no-slot, nested import.
• Added Error States (ES1–ES11): PulseEntry.DoesNotExist, extraction failure on submit (caught), extraction failure on /process/ retry (unhandled 500), partial item creation without rollback, missing evidence link, LLM provider errors, provider timeout, duplicate items on retry, /process/ evidence fallback, race condition, exception during pulse.save().

2026-06-12 — Provider switch: OpenAI out, Gemini + OpenRouter in

Type: Behavior Change

Summary:

• Replaced the declared-but-never-implemented OpenAI provider with apps/ai/providers.py exposing complete(system_prompt, user_prompt) behind get_provider() — MockProvider (default), GeminiProvider (REST generateContent), OpenRouterProvider (OpenAI-compatible REST).
• Settings: OPENAI_API_KEY removed; GEMINI_API_KEY/GEMINI_MODEL and OPENROUTER_API_KEY/OPENROUTER_MODEL added.

Tests:

• backend/tests/test_ai_providers.py — 14 tests.

2026-06-10 — Extraction quality: matched-answer titles, per-submission evidence, keyword classification

Type: Behavior Change

Summary:

• Item titles/descriptions now come from the specific answers_json entry that matched a keyword (trimmed to 120 chars), not raw_text[:100] of the whole pulse.
• extract_pulse_entry(pulse_id, evidence_id) now receives the evidence ID created by this submission.
• Risk classification by keyword group. Confidence reflects signal strength: single keyword 0.6, keyword in preferred question slot 0.75 (never 1.0).