Spec: F87 — Multimodal Chaos Capture (screenshot OCR · voice-note transcription · spreadsheet parsing)

Status: 🚧 built and contract-backed (backend + frontend shipped 2026-06-24; E2E describe added, make e2e not yet run green; Contract Graph migration 2026-07-14)

This completes the wedge. Lumi’s pitch is “Paste the mess. Get the brief.” — and the pitch is explicit that the mess is multimodal: “Zalo thread · Excel sheet · screenshot · voice note — any of it.” (pitch — “THE WEDGE” + “HOW IT WORKS”). Today F70 Chaos Import only accepts pasted chat text and typed notes. F70 §3 promises “screenshot (OCR’d to text), pasted Excel rows, voice-note transcript” but F70 §11 explicitly defers them: “Still not built: screenshot/voice-note upload path.” This spec owns the contract that turns a screenshot, a voice note, or a spreadsheet into text that flows into the existing F70 / F08 pipeline as a normal, evidence-linked source.

It builds no parallel system. It reuses F34 uploads for storage, F70 + F08 for extraction, EvidenceSource for provenance, the F37 provider layer for the Vietnamese-capable models, and defers the faithfulness invariant itself to F86. F87 is the capture-to-text stage that sits in front of all of them.

Contract Graph migration note (2026-07-14)

F87 is now declared in contracts/features/F87.cue. The contract owns the provider/parser boundary, tenant-scoped evidence and confidence handling, and processing endpoint lifecycle. Its executable backend proofs pass in the repository’s mock-default environment. This does not make the feature production-validated: live Gemini/provider calibration, consented multimodal field data, and browser/native runtime evidence remain explicit gaps in the generated graph and the implementation audit.

1. Problem & CEO Job

Capture with near-zero friction + see clearly. The buyer is an owner-operator of a Vietnamese field-service or specialty-contractor firm (HVAC/MEP, fit-out) whose real operating data lives in Zalo groups, Excel exports, and the owner’s memory (pitch — “THE BUYER”). In that world the owner’s morning input is almost never clean typed text: it is a screenshot of a Zalo thread, a voice note dictated from a job site, or an Excel sheet of jobs and deposits. If Lumi can only ingest text the owner types into a box, the wedge fails for the majority of real inputs — exactly the inputs the pitch promises Lumi accepts (“Ingests chat paste, screenshots, voice notes, spreadsheets” — pitch “EVIDENCE” slide, listed as built).

The reason this is viable now is cost (pitch — “WHY NOW”): “Structuring slang, typos, voice-note transcripts and screenshots in Vietnamese costs cents per interaction. Multimodal models only recently crossed the reliability and cost threshold that makes this viable for low-ACV businesses.” Without F87, the owner who lives in screenshots and voice notes must transcribe their own chaos before Lumi will read it — which is the manual labor Lumi exists to remove. With F87, the owner drops in whatever form the mess is already in and still gets today’s brief in under 10 minutes.

2. User Stories & Acceptance Criteria

• As an owner (first session, on a phone in the field), I want to upload a screenshot of a Zalo thread and get a brief, so that I do not retype my own chats.

  • Given a legible screenshot of a Vietnamese chat thread, when I upload it via the chaos-import surface, then the image is OCR’d to text, that text is extracted by the F70/F08 pipeline, and a Boss Brief renders with items each citing a span of the OCR’d text (not the raw image).
  • Given the OCR engine returns low confidence for the whole image (e.g. blurry photo), when processing completes, then the owner sees an honest “Chưa đọc rõ ảnh này — thử chụp lại hoặc gửi đoạn khác” state — never an invented brief built from guessed text.
  • Given OCR succeeds but a single region is unreadable, when items are produced, then the unreadable region lowers (never inflates) the confidence of any item derived near it, and items with no verbatim OCR-text span are dropped (F86 invariant).

• As an owner, I want to upload a voice note I recorded walking the site, so that my spoken updates become tracked commitments and blockers.

  • Given an audio file (m4a/mp3/ogg/wav) of a Vietnamese spoken update, when I upload it, then it is transcribed to text, the transcript flows into extraction, and produced items cite spans of the transcript.
  • Given transcription confidence is low (background noise, dialect), when items render, then the transcript confidence is surfaced on the source and folded into each item’s confidence; a whole-clip transcription failure yields the honest “Chưa nghe rõ — thử ghi lại” state, never a fabricated transcript.

• As an owner, I want to upload an Excel/CSV export of my jobs or receivables, so that the rows become structured signal.

  • Given a .xlsx or .csv file (or pasted tab/comma-separated rows), when I upload it, then it is parsed into normalized rows of text, those rows flow into extraction, and a produced item (e.g. “Client B deposit ₫50M — overdue”) cites the exact source row as its evidence span.
  • Given a row contains a payment/amount/date, when it is extracted, then F86’s no-invented-numbers rule holds against the parsed cell text — the number must appear verbatim in a parsed row or the item is dropped.

• As an owner, I want to see the upload being processed and then merge into the same brief, so that mixed inputs in one session produce one brief.

  • Given an upload, when processing is running, then the surface shows an honest processing state (skeleton, not a fake spinner) and resolves to the brief or an error within the <10-minute first-value budget (F70 §1).
  • Given a member of another org, when they request my upload’s processing status or extracted text, then they receive 404 — uploads and their extracted text never cross tenants.

• As any role, I want the same role/permission rules as the rest of Lumi, so that capture does not become a security hole.

  • Given a viewer role, when they attempt the presign/upload, then they receive 403 (inherited from F34’s IsOrgMember excluding viewer).

3. Scope Boundaries

• In:
  • (a) Screenshot / image → text — OCR of an uploaded image (PNG/JPG/WebP/HEIC), Vietnamese-capable.
  • (b) Voice note → transcript — ASR of an uploaded audio file (m4a/mp3/ogg/wav), Vietnamese-capable.
  • (c) Spreadsheet → structured rows — parse .xlsx/.csv, or pasted tab/comma-separated Excel rows, into normalized text rows.
  • The async processing pipeline that detects the upload’s kind, routes it to OCR / ASR / sheet-parse, produces normalized text + a per-source confidence, and feeds that text into the existing F70 / F08 extraction so produced items link an EvidenceSource of the correct type.
  • A provider-swappable service interface for OCR and ASR behind a single entry point (mirrors the F37 get_provider() pattern), with a documented default and honest cost/latency expectations.
  • Surfacing OCR/ASR confidence honestly and folding it into item confidence (the mechanism; the invariant is owned by F86).
• Out:
  • New storage. Reuse F34 pre-signed S3 POST + Upload model. F87 adds no bucket, no upload endpoint, no second storage path.
  • The extraction engine. Reuse F08 / F70. F87 produces text; it does not classify commitments/risks/decisions itself.
  • The faithfulness invariant. F86 owns “every item cites a verbatim span; never invent.” F87 only guarantees that the source text fed to extraction is the OCR’d / transcribed / parsed text, and that source confidence is surfaced.
  • Real-time / streaming audio (live dictation), video, multi-page PDFs, document layout analysis, handwriting recognition — all later, none in v1.
  • Voice-note capture in F32 Signal Inbox — F32 §3 explicitly lists “OCR, file uploads” as out; F32 §12 defers voice-note transcription. F87 is where OCR/ASR/sheet-parse now live; F32 may consume an F87-produced upload reference later, but that wiring is out of this spec.

4. Data Model

Reuses the F34 spine — extends the existing Upload model rather than adding a parallel one.

Reused (unchanged)

• Upload (F34, apps/uploads/models.py) — UUIDModel, organization FK (required), user FK, filename, content_type, byte_size, s3_key, status (pending → completed → expired), public_url, and evidence_source OneToOne to pulses.EvidenceSource. F34’s complete/ already creates the EvidenceSource.
• EvidenceSource (apps/pulses/models.py) — already has source_type with uploaded_file among its choices, plus content_text (the OCR’d / transcribed / parsed text lands here) and metadata_json (the per-source OCR/ASR confidence + provider lands here).
• ChaosImport (F70 §4) — already has input_kind ∈ {chat_paste, excel_paste, screenshot, voice_note, typed_note} and an EvidenceSource/Brief linkage. F87 supplies the text that the screenshot/voice_note/excel_paste kinds were always meant to carry.

Proposed additions (mark clearly — implemented under F87)

Extend Upload with the multimodal-processing fields (all nullable/defaulted so F34’s existing rows are untouched; Upload is already UUIDModel + tenant-scoped via organization FK):

# apps/uploads/models.py — F87 additions to Upload
PROCESSING_STATUS_CHOICES = [
    ("none", "None"),            # default — non-multimodal upload (existing F34 behavior)
    ("queued", "Queued"),        # async job enqueued
    ("processing", "Processing"),
    ("processed", "Processed"),  # extracted_text populated, confidence set
    ("failed", "Failed"),        # honest failure — see failure_reason
]

processing_status = models.CharField(max_length=12, choices=PROCESSING_STATUS_CHOICES, default="none")
modality          = models.CharField(max_length=12, blank=True)   # "" | "image" | "audio" | "spreadsheet"
extracted_text    = models.TextField(blank=True)                  # OCR'd / transcribed / row-flattened text
source_confidence = models.FloatField(null=True, blank=True)      # 0–1, engine-reported; NEVER 1.0, never hardcoded
processing_provider = models.CharField(max_length=40, blank=True) # which OCR/ASR/parser produced the text
failure_reason    = models.CharField(max_length=40, blank=True)   # "unreadable_image" | "no_speech" | "empty_sheet" | "provider_error"

• processing_status state machine (update backend/DIAGRAMS.md + GLOSSARY.md): none (F34 default) → on multimodal complete: queued → processing → processed | failed. failed is terminal until the owner retries (a retry re-enqueues to queued).
• EvidenceSource source subtypes. v1 keeps source_type="uploaded_file" (already in the enum, max_length=20) and records the precise modality in metadata_json ({"modality": "screenshot", "ocr_confidence": 0.74, "provider": "..."}) — no migration to the enum required. Proposed (optional, needs the max_length already covers it): add voice_note and screenshot as first-class SOURCE_TYPE_CHOICES only if the brief UI needs to badge them distinctly; until then metadata_json carries the subtype. Decision deferred to §12.
• Scale-check. No new table; Upload is already indexed by (organization, -created_at) and (organization, status). extracted_text is a TextField on an already-bounded per-upload row — no growth-with-usage join table, no unbounded count. Add an index on (organization, processing_status) for the status-poll query.

5. API Contract

F87 adds no new upload endpoint — uploading reuses F34’s presign → S3 POST → complete. F87 adds a processing trigger + status surface and teaches the F70 chaos-import endpoint to accept an upload reference.

• Tenant scoping (Rule 1): every query filters by organization_id=self.request.organization_id. The processing-status and chaos-import endpoints resolve upload_id only within the caller’s org; a cross-org id returns 404 (no existence confirmation, matching F34/F70). Celery tasks receive tenant_id: str and upload_id: str, never model objects (per backend/AGENTS.md).
• AI output / EvidenceSource (Rule 2): the produced text becomes EvidenceSource.content_text with source_type="uploaded_file" and metadata_json carrying {modality, source_confidence, provider}. Downstream items (Commitment/Risk/DecisionRequest/Task via F70/F08) each cite a verbatim span of that content_text. Honest confidence: the item confidence is the extraction confidence multiplied/ floored by the source confidence — low OCR/ASR confidence can only lower an item’s confidence, never raise it; never 1.0, never hardcoded. An item with no verbatim span in the extracted text is dropped (F86 invariant), not shown.
• Fail honest: on failed, the chaos-import endpoint refuses to fabricate a brief from a guessed transcript — it surfaces the honest error state (§9). Empty/garbage OCR or ASR output is treated as a low-signal failure, never auto-extracted.

Processing service interface (proposed — mirrors F37 get_provider())

# apps/uploads/processing.py (proposed)
class CaptureResult(NamedTuple):
    text: str
    confidence: float        # 0–1, engine-reported; clamped to a realistic band, never 1.0
    provider: str

def get_ocr_provider()  -> OCRProvider     # selected by settings.OCR_PROVIDER
def get_asr_provider()  -> ASRProvider     # selected by settings.ASR_PROVIDER
def parse_spreadsheet(blob, content_type) -> CaptureResult   # deterministic, in-process (openpyxl / csv)

• OCRProvider.recognize(image_bytes, language_hint="vi", mime_type=None) -> CaptureResult
• ASRProvider.transcribe(audio_bytes, language_hint="vi", mime_type=None) -> CaptureResult
• Both fail loud with a normalized CaptureProviderError (mirrors F37’s LLMProviderError); the Celery task catches it, sets processing_status="failed" + failure_reason, and never falls back to a fabricated transcript. A mock provider is the dev/CI default (ADR-002 pattern): deterministic text + a fixed sub-1.0 confidence, so the whole pipeline round-trips offline.
• gemini is now implemented (OCR_PROVIDER=gemini / ASR_PROVIDER=gemini): both modalities route through Gemini Flash’s generateContent endpoint via inline base64 data (mime_type from the upload’s content_type), reusing F37’s exact plumbing and the single GEMINI_API_KEY. Confidence is the model’s self-assessed legibility/clarity, clamped into [0.50, 0.90] (never 1.0). Empty text → unreadable_image / no_speech; network/5xx → TransientCaptureError (retried); 4xx/malformed → permanent CaptureProviderError. See ADR-028. Chirp 3 / Cloud STT (ASR_PROVIDER=google_stt) is the documented future opt-in behind this same ASRProvider interface.

6. UI / UX

• Route / page: the existing F70 /intel/import ChaosImportPage (ChaosImportInput) gains capture affordances; no new route. The shared F34 useUpload hook + FileAttachButton are the integration surface (presign → S3 → complete), extended with the modality types.
• Affordances (mobile-first — this is field use, DESIGN.md + TASTE.md):
  • Image — “Chụp / tải ảnh màn hình” (camera capture on mobile via <input capture>, gallery on desktop).
  • Audio — “Ghi âm / tải voice note” (in-browser record button + file upload fallback).
  • Spreadsheet — “Tải Excel / CSV” (file picker) and the existing paste box already accepts pasted Excel rows (input_kind="excel_paste").
• States to design: empty (paste/upload prompt with owner-vocabulary placeholder), processing (honest skeleton per upload with the modality label — “Đang đọc ảnh…” / “Đang nghe voice note…” / “Đang đọc bảng…”, polled via GET /uploads/{id}/processing/, not a fake spinner), populated (the upload’s extracted items merge into the same Boss Brief preview with evidence badges + a source-confidence cue), low-confidence / failed (honest “chưa đọc/nghe rõ” with a Thử lại retry calling reprocess/), error, no-permission (viewer cannot upload).
• Mixed-input merge: multiple uploads + a paste in one session resolve into one brief; each item’s evidence badge names its origin (screenshot / voice note / row), reusing F70’s sectioned brief (Việc cần xử lý / Cần chốt với bên ngoài / Rủi ro / Tin nhắn nên gửi).
• shadcn components (check frontend/src/components/ui/ first): Button, Card, Badge (confidence + modality), Skeleton (processing), Tabs (input kinds, already in F70), Progress (upload/record). No raw Tailwind, no stock palette — Lumi semantic tokens only (Rule 4).
• i18n (Rule 3): extend the F70 chaosImport namespace with F87 keys in both en/ and vi/ (Vietnamese written first per F18): capture_image, capture_audio, capture_sheet, processing_image, processing_audio, processing_sheet, low_confidence_image, low_confidence_audio, error_unreadable_image, error_no_speech, error_empty_sheet, source_confidence_label, retry_processing. Owner vocabulary, never “AI” or “OCR/ASR” in the owner-facing copy.

7. Test Plan

Backend coverage must stay ≥97% (CLAUDE.md) / ≥99% where the surrounding suite requires; providers always mocked (deterministic). New E2E describe block required for this user-facing flow.

8. Current Behavior

• F34 upload exists and is shipped — apps/uploads/ (Upload model, /api/uploads/presign/, /api/uploads/{id}/complete/, S3 key + CDN URL, EvidenceSource(source_type="uploaded_file") linkage), plus the useUpload hook + FileAttachButton on the frontend.
• OCR / ASR / spreadsheet-parse are not built. Grep over backend/ finds no Whisper/vision/OCR/ transcription code and no openpyxl dependency; the only source_type writes are existing uploaded_file/external_event/manual_note paths.
• F70 defers them. F70 §11 — “Still not built: screenshot/voice-note upload path.” F70 §12 lists “Screenshot OCR — which service/model” and “Voice-note transcription — source and language handling (defer to F37)” as open questions. The F70 ChaosImport.input_kind enum already reserves screenshot, voice_note, and excel_paste, but no code produces text for them.
• F32 Signal Inbox is text-only by design (§3 Out: “OCR, file uploads”; §12 defers voice-note).
• Celery is available (celery>=5.4, config/celery.py, @shared_task with autoretry_for/retry_backoff already used in apps/briefs/tasks.py and apps/followups/tasks.py; tests run CELERY_TASK_ALWAYS_EAGER).
• The F37 provider abstraction exists (apps/ai/providers.py, get_provider(), LLM_PROVIDER, mock default) — the exact pattern F87’s OCR/ASR provider layer must mirror.

This is a new capture stage; no F87 behavior exists today.

9. Error States

• Retry: the async job uses Celery autoretry_for/retry_backoff (mirroring apps/briefs/tasks.py) for transient provider errors only; a genuine unreadable input is not retried automatically — it surfaces the honest failure with a manual Thử lại. Retries respect the <10-min first-value budget.
• Never silently swallow a processing error, never fabricate text to fill the brief, never inflate item confidence to mask a low-confidence source.

10. Diagrams & Docs Touched

• DIAGRAMS.md: AI pipeline — add the capture → OCR/ASR/parse → text → F70 extraction stage in front of the paste→brief path.
• backend/DIAGRAMS.md: Upload.processing_status state machine (none → queued → processing → processed | failed); the async processing request lifecycle (complete → enqueue → process → chaos-import).
• frontend/DIAGRAMS.md: /intel/import state set gains the per-upload processing/failed/retry states (no new route).
• GLOSSARY.md: add OCR, ASR / transcription, transcript confidence / source confidence, modality.
• ADRs: depends on ADR-002 (mock-default provider pattern); a new ADR is required only if a single OCR/ASR vendor is locked in (see §12).

11. Implementation Audit

Built (2026-06-24)

Shipped — backend + frontend. F87 was implemented on the F34/F70/F37 spine exactly as specified; the async capture stage now turns screenshots, voice notes, and spreadsheets into evidence-linked text that flows into the existing chaos-import/extraction pipeline. What shipped:

• Model (apps/uploads/models.py) — Upload extended with processing_status (none → queued → processing → processed | failed, default none), modality (image | audio | spreadsheet), extracted_text, source_confidence (0–1, never 1.0), processing_provider, failure_reason (unreadable_image | no_speech | empty_sheet | provider_error). New index (organization, processing_status). No new table, no new bucket.
• Processing layer (apps/uploads/processing.py) — CaptureResult NamedTuple; get_ocr_provider() / get_asr_provider() selected by settings.OCR_PROVIDER / settings.ASR_PROVIDER with a deterministic mock provider as the dev/CI default (mirrors F37 / ADR-002); parse_spreadsheet() deterministic in-process via openpyxl/csv; normalized CaptureProviderError.
• Async task (apps/uploads/tasks.py) — @shared_task process_upload(tenant_id, upload_id) enqueued from complete/ for image/audio/spreadsheet content types; routes to OCR/ASR/parse; writes extracted_text + source_confidence and sets EvidenceSource.content_text + metadata_json {modality, source_confidence, provider}. Fail-honest — sets failure_reason, never fabricates text or inflates confidence. Takes tenant_id+upload_id strings, never model objects.
• Endpoints — GET /api/uploads/{id}/processing/ (tenant-scoped status poll), POST /api/uploads/{id}/reprocess/ (re-enqueue a failed upload to queued); complete/ response extended with processing_status + modality; POST /api/chaos-import/ accepts upload_id and uses upload.extracted_text (400 not-yet-processed, 409 failed, 404 cross-org).
• Confidence folding — item confidence is the extraction confidence floored/multiplied by source_confidence (lower-only, never 1.0). F86 faithfulness holds against extracted_text; unsourced items dropped.
• Frontend — /intel/import ChaosImportPage gains image/audio/spreadsheet capture affordances; honest per-upload processing skeleton polled via /uploads/{id}/processing/ with TanStack Query; failed/retry (reprocess/) state; items merge into the same Boss Brief. New chaosImport i18n keys in en/ + vi/.
• Tests — backend cases per §7 (OCR/ASR/sheet-parse, fail-honest, cross-tenant isolation, faithfulness against extracted text, confidence folding, chaos-import upload_id 400/409, viewer 403, reprocess); frontend ChaosImportPage processing/retry; E2E describe("Multimodal Chaos Capture") added (mocked S3 + mock providers).
• Dependency — openpyxl added to backend/pyproject.toml, installed via uv.

Outstanding (honest): make e2e was not run green in the implementing session — the E2E describe block exists but the full Playwright suite has not yet been verified end-to-end (MVP_SCOPE E2E cell stays 🔲). The OCR/ASR vendor defaults remain mock; a production vendor lock-in still requires the ADR per §12.

Original constraints (at spec creation) — preserved

• Reuse F34 storage — apps/uploads/Upload + presign/complete already exist and create the EvidenceSource. F87 extends Upload, it does not add a second upload path.
• Reuse the F70 / F08 pipeline — produced text must enter run_chaos_import() / F08 extraction as a normal source; do not fork extraction. The ChaosImport.input_kind enum already reserves the modalities.
• Reuse the F37 provider pattern — OCR/ASR providers behind get_*_provider() with a mock default and CaptureProviderError; never hardcode a vendor in call sites.
• F86 faithfulness must hold against the OCR’d/transcribed text — the source text becomes the evidence substrate; F87 surfaces source confidence and folds it into item confidence (lower-only), but the no-invented-span invariant itself is owned by F86. (F86 may not exist yet at spec-creation time — referenced by ID.)
• Async must respect the <10-min first-value budget (F70 §1) — Celery is already wired (CELERY_TASK_ALWAYS_EAGER in tests, @shared_task with retry/backoff in apps/briefs/apps/followups); tasks take tenant_id+upload_id strings, never objects.
• New dependency: openpyxl (spreadsheet parsing) must be added to backend/pyproject.toml and installed via uv (CLAUDE.md). OCR/ASR are remote provider calls — no heavy local model dependency.

12. Open Questions / Decisions

• OCR provider + cost. RESOLVED (ADR-028, 2026-06-25). Default (when enabled) to Gemini Flash (gemini-2.5-flash) via generateContent — a Vietnamese-capable multimodal model sharing the F37 provider plumbing and the single GEMINI_API_KEY, ~$0.0004/image (“cents per interaction”). One provider surface, strong Vietnamese diacritics. A dedicated-OCR escape hatch remains available behind the same OCRProvider interface. Mock stays the dev/CI default.
• ASR provider + cost. RESOLVED (ADR-028, 2026-06-25). Default (when enabled) to the same Gemini Flash model via generateContent inline audio — one key, Vietnamese-capable, ~$0.06/hour of audio (field voice notes <60s ⇒ cents). Chirp 3 / Cloud Speech-to-Text (ASR_PROVIDER=google_stt) is the documented future opt-in behind the same ASRProvider interface for calibrated word-level confidence or streaming — not built now (it needs GCP service-account auth, a different auth model than the GEMINI_API_KEY API key).
• EvidenceSource subtype. Keep uploaded_file + metadata_json.modality, or promote screenshot / voice_note to first-class SOURCE_TYPE_CHOICES? Recommendation: metadata_json for v1 (no migration); promote only if the brief UI needs distinct badges. (max_length=20 already accommodates both strings.)
• Sync vs async UX. A spreadsheet parse is fast and in-process (could be synchronous); OCR/ASR are remote and slow. Recommendation: a single async path for all modalities for UI consistency (one processing state), with spreadsheet parse simply resolving fastest. Revisit if the async hop hurts the spreadsheet first-value time.
• Confidence band. RESOLVED (partial). Real Gemini OCR/ASR confidence is the model’s self-assessed legibility/clarity, clamped into [0.50, 0.90] (missing → floor) — a proxy, not a calibrated engine score (ADR-028). It folds lower-only via source_confidence. Final per-item folding (multiply vs floor) is still to be tuned against the F70 G5 / F86 benchmark.

13. Bug / Change Log

2026-06-24 — Spec created

Type: Feature

Summary:

• Created F87 to own Lumi’s multimodal capture contract — screenshot OCR, voice-note transcription, and spreadsheet parsing — the inputs the pitch (“Paste the mess. Get the brief.”) promises and that F70 §3 named but §11 deferred (“Still not built: screenshot/voice-note upload path”).

Reason:

• The wedge’s input is explicitly multimodal (pitch — “THE WEDGE”: “Zalo thread · Excel sheet · screenshot · voice note — any of it”; “EVIDENCE”: “Ingests chat paste, screenshots, voice notes, spreadsheets”). Today only chat-paste and typed-note work. F87 closes that gap by turning each modality into text that flows into the existing F70/F08 pipeline as a normal evidence-linked source — no parallel ingestion system.

Impact:

• New draft feature F87 in the registry. Reuses F34 storage, the F70/F08 extraction pipeline, the F37 provider pattern, EvidenceSource provenance, and defers the faithfulness invariant to F86. Extends the existing Upload model with processing_status/modality/extracted_text/source_confidence; adds no new storage. No code shipped by this spec.

Tests:

• Test plan defined (§7); none implemented yet.

Notes:

• Provider-swappable, cost-aware (cents per interaction), Vietnamese-capable by default; mock provider is the dev/CI default per ADR-002. Async processing must respect F70’s <10-min first-value budget. Fail honestly (“couldn’t read this image” / “couldn’t hear this”) — never invent text or inflate confidence.

2026-06-24 — Implemented F87

Type: Feature

Summary:

• Built the multimodal capture stage end to end on the F34/F70/F37 spine: extended Upload with processing_status/modality/extracted_text/source_confidence/processing_provider/failure_reason (+ (organization, processing_status) index); added apps/uploads/processing.py (CaptureResult, get_ocr_provider()/get_asr_provider() with mock defaults, deterministic parse_spreadsheet(), CaptureProviderError) and apps/uploads/tasks.py (@shared_task process_upload). New endpoints GET /uploads/{id}/processing/ and POST /uploads/{id}/reprocess/; complete/ returns processing_status+modality; chaos-import/ accepts upload_id. /intel/import gained image/audio/ spreadsheet capture affordances with an honest polled processing skeleton and failed/retry state.

Reason:

• Closes the deferred half of the wedge — F70 §11 “Still not built: screenshot/voice-note upload path.” The pitch promises Lumi ingests “Zalo thread · Excel sheet · screenshot · voice note — any of it”; until now only chat-paste/typed-note worked. F87 turns each modality into evidence-linked text that flows into the existing extraction pipeline, so the owner drops in whatever form the mess is already in.

Impact:

• F87 flips to 🚧 in the registry (Backend ✅, Frontend ✅, E2E 🔲). No new table, no new bucket — extends F34 Upload and reuses F70/F08 extraction, EvidenceSource provenance, and the F37 mock-default provider pattern. Confidence is folded lower-only by source_confidence (never 1.0); F86 faithfulness holds against extracted_text; failures are honest (unreadable_image/no_speech/empty_sheet/ provider_error), never fabricated. New dependency: openpyxl.

Tests:

• Backend §7 cases implemented (OCR/ASR/sheet-parse, fail-honest, cross-tenant isolation, faithfulness against extracted text, confidence folding, chaos-import upload_id 400/409, viewer 403, reprocess); frontend ChaosImportPage processing/retry; E2E describe("Multimodal Chaos Capture") added. Backend coverage held ≥97%. make e2e not yet run green this session — E2E cell stays 🔲 until verified.

Notes:

• OCR/ASR vendor defaults remain mock providers; locking a single production vendor still requires an ADR (§12). Diagrams updated: DIAGRAMS.md §4a (capture→OCR/ASR/parse→text→F70), backend/DIAGRAMS.md §23a (Upload processing state machine) + §23b (async lifecycle), frontend/DIAGRAMS.md §6a (per-upload processing/failed/retry states), GLOSSARY.md (OCR, ASR/transcription, source confidence, modality, Upload.processing_status).

2026-07-02 — Spec drift correction: check all shipped ACs against §11 audit

Type: Documentation

Summary:

• Corrected all 10 acceptance criteria from [ ] to [x] based on §11 Implementation Audit confirmation.
• Criteria checked: (1) image OCR → extraction → brief with OCR-text citations; (2) low OCR confidence → honest unreadable_image failure state; (3) partial unreadable region lowers item confidence, unsourced items dropped (F86); (4) audio ASR → transcript → extraction → citations; (5) low transcription confidence surfaced, whole-clip failure → honest no_speech state; (6) xlsx/csv → normalized rows → extraction → source row citation; (7) F86 no-invented-numbers holds against parsed cell text; (8) honest processing skeleton (TanStack Query poll, not fake spinner); (9) cross-tenant 404 for upload/extracted-text/processing-status; (10) viewer role → 403 at presign.

Evidence: §11 “Shipped — backend + frontend” (2026-06-24); model fields, processing layer, async task, endpoints, confidence folding, frontend, and tests all confirmed ✅; §11 backend §7 cases listed explicitly (OCR/ASR/sheet-parse, fail-honest, cross-tenant isolation, faithfulness, confidence folding, upload_id 400/409, viewer 403, reprocess); 2026-06-24 and 2026-06-25 change log entries confirm scope.

2026-06-25 — Wired Gemini Flash OCR + ASR providers

Type: Feature

Summary:

• Extended the F87 provider interface with real Gemini Flash OCR + ASR behind the existing swappable selectors. Added GeminiOCRProvider/GeminiASRProvider to apps/uploads/processing.py: both POST a multimodal generateContent request (inline base64 data + responseMimeType=application/json) mirroring apps/ai/providers.py::GeminiProvider exactly — same URL, x-goog-api-key header, 30s timeout. Added an optional mime_type kwarg to recognize()/transcribe() on the mock + new providers; process_upload passes mime_type=upload.content_type. get_ocr_provider()/get_asr_provider() now accept mock | gemini. Settings gained OCR_MODEL/ASR_MODEL (default GEMINI_MODEL); OCR_PROVIDER/ ASR_PROVIDER defaults unchanged (mock).

Reason:

• Resolves the §12 OCR/ASR provider open questions (ADR-028): one key, Vietnamese-capable, cents/interaction, reuses F37 plumbing — without GCP service-account auth. Chirp 3 / Cloud STT recorded as the documented future opt-in (ASR_PROVIDER=google_stt).

Impact:

• F87 stays 🚧 (Backend ✅); no registry status change, no frontend change. Mock stays the dev/CI default (ADR-002); real providers are opt-in and never silently fall back to mock. Confidence is the model’s self-assessed value clamped into [0.50, 0.90] (never 1.0). Fail-honest preserved: empty text → unreadable_image/no_speech; network/5xx → TransientCaptureError (retried); 4xx/malformed → permanent CaptureProviderError.

Tests:

• tests/test_uploads.py: request-shape+parse, mime defaulting, confidence clamp (high/low/missing), empty-text fail-honest, 5xx→transient, ConnectionError→transient, 4xx/empty-candidates/non-JSON/ non-string→permanent, constructor-without-key, gemini selectors, and a task-level test that process_upload with OCR_PROVIDER=gemini writes extracted_text + folds confidence. HTTP always mocked. apps/uploads/processing.py at 100% line coverage.