Spec: F33 — Connected Source Signals

Status: partial under Contract Graph — backend apps/sources, SourcesPage.tsx, tests, and E2E markers exist; browser/live-provider evidence remains explicit. Contract: F33.cue · Audit: F33 audit · Test plan: F33 plan Reference study: current official product surfaces from Slack AI, Notion AI, Glean, Asana AI, Microsoft 365 Copilot, Atlassian Rovo, and HubSpot Breeze — summarized in §11.

1. Problem & CEO Job

See clearly + trust the signal. Human-reported truth is necessary but insufficient. If nobody reports a problem, Lumi must still know; otherwise it is advisory theater.

F07, F21, F29, and F32 capture human signals. F33 adds machine truth from systems of record: support queues, work trackers, accounting, CRM, payments, and other apps where evidence appears before a human summarizes it. Lumi does not clone them; it ingests exceptions and changes into the executive control loop.

That is the difference between a status tool and an operating system.

2. User Stories & Acceptance Criteria

• As an admin, I want to connect a source system once, so that Lumi starts receiving the operational facts already living elsewhere.
  • Given a supported source, when I connect it, then Lumi shows connection status, last successful sync, and the categories of signals it can emit.
  • Given a broken token, webhook secret, or permission issue, then the connection is visibly degraded — silent connector failure is a trust bug.
• As a CEO, I want source-system exceptions to arrive in Lumi as reviewable signals, so that I no longer depend only on what people remember to report.
  • Given a mapped source event (for example overdue invoice, VIP ticket escalation, slipped milestone, failed payment, stalled deal), then Lumi records a SourceEvent, creates an EvidenceSource, and, if it passes the signal rule, opens a Signal Inbox item (F32) with capture_channel="connector".
  • Source events are deduplicated by connection + external event id; replayed webhooks or repeated sync windows must not create duplicate executive noise.
• As a manager, I want the same review flow for connector signals as for manually captured signals, so that Lumi keeps one queue rather than parallel queues.
  • Any connector-origin signal that deserves human review lands in the same F32 inbox and uses the same accept/reject/promotion mechanics.
  • Connector signals never bypass human review to mutate live tasks, risks, commitments, decisions, or approvals in v1.
• As a CEO, I want source signals to strengthen the Brief instead of flooding it, so that more data means more trust, not more noise.
  • Only accepted/promoted connector signals enter the operational objects that feed Brief/Today.
  • Extremely high-severity events may surface as visible inbox urgency, but still do not auto-dispatch work without a human accepting them.
• As a global product, I want a source-framework rather than one-off hacks, so that Lumi can support different tool stacks without becoming an integration consulting project.
  • v1 supports a connector contract and three high-value categories:
    • work systems (Jira / Linear / Asana class)
    • support/customer systems (Zendesk / Intercom / Freshdesk class)
    • cash/revenue systems (QuickBooks / Xero / Stripe / NetSuite class)
  • A generic inbound webhook connector exists so an ops team can push events from any other system before a native adapter is built.

3. Scope Boundaries

• In: source connection records, secure auth/secrets, polling/webhook ingestion, normalized source events, signal rules, dedupe, health/status UI, F32 inbox handoff, provenance, category-first connector framework, generic webhook ingestion.
• Out: full mirror UIs of source systems, full bi-directional sync, automatic write-back into source systems, “connect 100 apps on day one”, ambient employee surveillance, email/calendar sync as knowledge-management products, generic enterprise-search scope creep.

4. Data Model

• New app sources.
• SourceConnection — org FK, provider slug, category (work | support | cash | crm | custom_webhook), status (active | degraded | disabled), display_name, auth/config encrypted, last_synced_at, last_error_text, cursor_json, created_by.
• SourceRule — org FK, connection FK nullable (org-wide defaults allowed), event_type, threshold_json, severity_mapping_json, proposal_type target (task | commitment | risk | decision | approval | none), enabled.
• SourceEvent — org FK, connection FK, external_event_id, event_type, external_entity_type, external_entity_id, occurred_at, title, body_text, source_url, payload_json, normalized_hash, severity_hint, status (captured → routed | ignored | failed), evidence_source FK, signal_inbox_item FK nullable. Unique (connection, external_event_id).

F32 glue:

• SignalInboxItem.capture_channel expands from web | telegram to web | telegram | connector.

Evidence:

• EvidenceSource.source_type must support external_event (or another equally explicit name decided in the implementing PR). Using manual_note here would blur provenance and is wrong.

5. API Contract

Promotion path:

• No dedicated promote endpoint lives here. Promotion happens through F32 once a SourceEvent has opened a SignalInboxItem.

Tenant scoping:

• Every connection, rule, event, evidence source, and inbox item is scoped to request.tenant.
• Shared enterprise connectors must not leak data across orgs even when one vendor account contains multiple workspaces/companies.

6. UI / UX

• New route: /settings/sources (admin-only).
• Two surfaces only:
  • Connections: status cards, connect/reconnect, last sync, degraded warning.
  • Recent source events: recent captured events with whether they became inbox items.
• This is not an admin labyrinth. If the UI starts looking like an iPaaS product, it is wrong.
• Signal review does not happen here. It happens in /intel/inbox with the same F32 review flow.
• i18n: new sources namespace, en + vi.

7. Test Plan

8. Current Behavior

Source-system ingestion is shipped. apps/sources provides SourceConnection, SourceRule, and SourceEvent; generic webhook ingest routes into F32’s Signal Inbox through configured rules; /settings/sources exposes connection health and recent events in the web UI.

Contract Graph boundary: repository-controlled backend and web-unit proofs are registered, but browser execution, live provider delivery, and production evidence are not claimed from source or proof collection.

9. Error States

• sources.error_invalid_config — admin entered invalid connector config
• sources.error_bad_signature — webhook signature/secret invalid
• sources.error_duplicate_event — event replay acknowledged but ignored safely
• sources.error_sync_failed — sync attempt failed; connection becomes degraded
• sources.error_mapping_failed — event captured, but could not be normalized or routed; event remains visible for diagnosis

Operational rule:

• Connector failures must be visible in the UI and auditable. Silent stale sync is the worst failure mode.

10. Diagrams & Docs Touched

• DIAGRAMS.md capture loop expands to human + system signals
• backend/DIAGRAMS.md sources ER diagram
• frontend/DIAGRAMS.md /settings/sources route
• F32 spec (connector-origin inbox items)
• F14 spec (future grounded answers over connected sources)

11. Implementation Audit (2026-06-12)

Edge Cases Catalog (formerly §9, 2026-06-14; 4 items): duplicate delivery idempotency (idempotent across retries/webhook replays/sync retries), rules pointing to deleted entities failing closed with diagnostics, out-of-order events preserving both ingestion order and source event time, high-volume burst isolation (one connector must not starve other tenants). Full catalog in git history.

Shipped. Native vendor-specific adapters beyond generic webhook and auto-promotion into operational objects remain deliberately out of scope.

12. Open Questions / Decisions

• Which exact first native connectors ship after custom_webhook? Recommendation: one from each category in §2, chosen by current design-partner pain rather than logo prestige.
• Should source rules support per-team routing in v1? Recommendation: yes for destination owner/team defaults, but no visual rule builder beyond what an admin can understand in five minutes.

13. Bug / Change Log

2026-07-02 — §11 Row 5 frontend path corrected (batch 12)

Type: Documentation Truth Fix

Summary:

• §11 Row 5: Corrected SourcesPage.tsx path from frontend/src/pages/SourcesPage.tsx (old pre-IA-restructure location, no longer exists) to frontend/src/pages/settings/SourcesPage.tsx (verified by ls).

2026-07-02 — Spec drift fix: check shipped ACs against §11 audit

Type: Documentation Truth Fix

Summary:

• Checked all 10 previously open [ ] ACs in §2 that §11 confirms as implemented:
  • AC1 (connection status + last sync + signal categories visible) — rows 1, 2, 5 (SourceConnection model + /api/source-connections/ + SourcesPage.tsx)
  • AC2 (broken token/secret → visibly degraded) — rows 1, 2, 4 (degraded status + test coverage); §9 degraded error state
  • AC3 (mapped source event → SourceEvent + EvidenceSource + SignalInboxItem connector) — row 7 (direct match)
  • AC4 (dedup by connection + external_event_id) — rows 1, 4 (unique constraint on model + test case 2)
  • AC5 (connector signals land in same F32 inbox) — row 7 (direct match)
  • AC6 (connector signals never bypass human review) — row 7 (routes to inbox, not directly to operational objects)
  • AC7 (only accepted connector signals enter operational objects) — rows 1, 7 (no auto-promote path)
  • AC8 (high-severity events → inbox urgency but no auto-dispatch) — rows 1, 7 (severity_hint on SourceEvent; all events route through F32 inbox)
  • AC9 (connector contract + 3 high-value categories) — row 1 (SourceConnection.category: work|support|cash|crm|custom_webhook)
  • AC10 (generic inbound webhook connector) — row 3 (direct match: /api/source-connections/webhook//)

2026-06-21 — Rename source connections

Type: Feature (CRUD completion)

Summary: SourceConnectionViewSet allows patch, but the UI exposed only connect/disconnect/poll/delete — a connection’s display_name couldn’t be edited. Added sourcesApi.updateConnection + a Rename dialog (display_name input → PATCH) per ConnectionCard in SourcesPage.tsx. i18n keys added to en/vi (then propagated). testids connection-edit-{id}/connection-edit-save.

2026-06-15 — Frontend route namespaced

Frontend route namespaced: /sources -> /settings/sources and /inbox -> /intel/inbox (app-wide IA restructure; API paths unchanged).

2026-06-14 — Spec restructured to TEMPLATE.md canonical order (§1–§13)

Type: Documentation Change

Summary:

• §9 Edge Cases (4 items) absorbed into §11 Implementation Audit EC note.
• §10 Error States → §9, §11 Diagrams & Docs Touched → §10, §12 Implementation Audit → §11 (canonical 4-col table, 7 rows).
• §13 Reference Study → §14 Reference Study (Appendix); §14 Open Questions → §12; §15 Bug / Change Log → §13.

Reason:

• Align with TEMPLATE.md canonical section order per SP fleet restructuring pass.

Impact:

• Spec authors and future implementation agents.

2026-06-12 — Spec sync after ship audit

Type: Doc Fix

Summary:

• Added the missing Implementation Audit section and replaced the stale pre-ship Current Behavior text with the shipped generic-webhook state.

Reason:

• The registry, plan, code, and tests all showed F33 as shipped, but the spec still described the feature as nonexistent and had no audit section.

14. Reference Study (Appendix, 2026-06-11)

This feature exists because the current market leaders converge on the same structural idea:

Observed from official product docs/pages:

• Slack AI emphasizes daily briefings, summaries, enterprise search, and searching integrated app data from Google Drive, Salesforce, tickets, and files.
• Notion AI emphasizes enterprise search across connected apps, AI meeting notes, and agents that act with workspace + connected-app context.
• Glean centers on connectors, real-time sync, permission inheritance, and actions with human-in-the-loop controls.
• Asana AI centers on AI teammates, morning priorities, connectors, MCP, and workflow automation across work systems.
• Microsoft 365 Copilot centers on a shared intelligence layer over emails, files, meetings, chats, and transactions, with agents and governed access.
• Atlassian Rovo centers on search, chat, agents, and a teamwork graph across Atlassian + connected SaaS apps.
• HubSpot Breeze centers on AI grounded in CRM data, conversations, deals, and next-best actions.

Adopted here:

Deliberately rejected:

Impact:

• F33 now has a complete traceability chain: spec → plan → tests → audit.

Tests:

• No code changes. Verification comes from backend/tests/test_sources.py and frontend/tests/e2e/sources.spec.ts.

Notes:

• Native adapters remain future work by design; the shipped generic webhook path is the current contract.

2026-06-11 — Initial spec

Type: Feature

Summary:

• Added Connected Source Signals as the system-of-record ingestion layer for Lumi’s executive control loop.

Reason:

• Without machine truth from external systems, Lumi still depends too much on human self-reporting to earn global trust.

Impact:

• Defines connection management, normalized source events, signal rules, and F32 inbox handoff.

Tests:

• Planned in backend/tests/test_sources.py plus Signal Inbox regressions.

Notes:

• This is a wedge feature, not a generic integrations program.