partial — configuration, tracing foundation, redaction, Collector wiring, product events, and dashboard definitions exist; backend runner and deployed seven-day validation remain open

Implement OpenTelemetry observability with PostHog as product-debug backend

Status: partial — configuration, tracing foundation, redaction, Collector wiring, product events, and dashboard definitions exist; backend runner and deployed seven-day validation remain open

Goal

Implement OpenTelemetry-based observability for Lumi, with PostHog as the first product-debug backend — not as a hard-coupled, permanent infrastructure APM dependency.

The system should let us answer, fast:

  • Which tenant/user hit this failure?
  • What source input, AI workflow, feature flag, request, worker job, and backend error were involved?
  • Was the failure product-visible, silent, retried, corrected, ignored, or user-blocking?
  • Can we jump from a backend error/log/trace to the relevant PostHog user/session/event context?
  • Can we later fork telemetry to another backend without reinstrumenting Lumi?

This is not vanity observability. For Lumi, trust depends on source-linked outputs, no hallucinated commitments, safe AI action boundaries, and fast debugging of messy real-world input.

External facts to preserve

As of 2026-07-07:

  • PostHog Logs supports OTLP/OpenTelemetry logs.
  • PostHog says standard OpenTelemetry libraries can send logs to its HTTP endpoint using the project token; no vendor-specific SDK is required.
  • PostHog Distributed Tracing supports OTLP spans/traces, but the feature is marked alpha.
  • OpenTelemetry Collector is vendor-agnostic and can receive, process, and export telemetry to one or more backends.

Implication: use OpenTelemetry as the instrumentation standard, PostHog as the first backend, and the Collector as the anti-lock-in layer.

Non-goals

  • Do not send raw customer content to PostHog.
  • Do not log full prompts, full chat messages, full emails, full uploaded files, raw OCR text, invoices, phone numbers, addresses, or payment data.
  • Do not make PostHog the only possible observability destination.
  • Do not build Grafana/Prometheus/Loki/Tempo in this issue unless validation proves PostHog cannot answer basic product-debug questions.
  • Do not add fake security, compliance, or data-residency claims to docs or marketing.

Target architecture

Lumi frontend / mobile / backend API / worker jobs / AI workflows
        ↓
OpenTelemetry SDKs + structured application logging
        ↓
OpenTelemetry Collector
        ↓
 ┌──────────────────────────────┬──────────────────────────────┐
 │ PostHog                       │ Optional future backends      │
 │ product analytics             │ Prometheus/Grafana metrics    │
 │ feature flags                 │ Loki/Better Stack logs        │
 │ session replay                │ Tempo/Jaeger/Datadog traces   │
 │ errors/logs/traces/AI signals │ Sentry error detail           │
 └──────────────────────────────┴──────────────────────────────┘

Direct-to-PostHog is acceptable only for a local spike. Production should prefer Collector routing so we can batch, retry, redact, sample, and fork signals later.

Signals to capture

1. Backend request traces

Capture spans around:

  • HTTP request lifecycle
  • auth and tenant resolution
  • DB query hotspots
  • external API calls
  • file upload / OCR / parse pipeline boundaries
  • AI extraction, summarization, classification calls
  • task/risk/commitment creation flows
  • notification / reminder / daily brief generation
  • background worker jobs

Minimum span/log attributes:

service.name
service.version
deployment.environment
trace_id
span_id
request_id
tenant_id_hash
user_id_hash
route
method
status_code
duration_ms
error.type
error.message_sanitized
feature_flag_keys
worker_job_type
queue_latency_ms
retry_count

2. Product-debug events

Emit PostHog events for user-visible state transitions, not noisy internals:

input_submitted
source_parsed
ai_extraction_started
ai_extraction_completed
ai_extraction_failed
assistant_response_completed
commitment_drafted
risk_drafted
task_drafted
user_accepted_ai_output
user_corrected_ai_output
user_ignored_ai_output
daily_brief_generated
daily_brief_opened
stale_item_chased
undo_used

Each event should include correlation IDs:

trace_id
request_id
tenant_id_hash
user_id_hash
source_id
workflow_id
ai_run_id
feature_flag_variant

3. AI workflow traces

For every AI run, capture structured metadata without raw private content:

ai_run_id
workflow_type              # parse_voice_note | parse_chat_paste | parse_excel | daily_brief | risk_detection
model_provider
model_name
input_token_count
output_token_count
latency_ms
cost_usd_estimate
source_type                # voice_note | pasted_chat | screenshot | excel | manual_note
source_language
source_char_count_bucket
confidence_bucket
output_item_count
action_boundary            # draft_only | suggest_only | allowed_action
user_outcome               # accepted | corrected | ignored | expired | undone
failure_class              # provider_error | timeout | invalid_schema | low_confidence | policy_block | redaction_block

Do not capture:

raw_prompt
raw_completion
raw_source_text
customer names
customer phone numbers
addresses
invoice/payment identifiers
uploaded file contents

4. Metrics / health

PostHog can be the first observability UI, but do not assume it is the long-term metrics backend.

Track at least:

request_count
request_error_rate
request_duration_p50/p95/p99
worker_success_rate
worker_retry_rate
ai_run_success_rate
ai_run_latency_p50/p95/p99
ai_cost_usd_daily
brief_generation_success_rate
accepted_vs_corrected_ratio
redaction_drop_count
otel_export_failure_count

If PostHog cannot give usable p95/error-rate dashboards or alerting, add a small metrics backend separately instead of forcing PostHog to be something it is not.

Privacy and redaction requirements

Create a telemetry sanitizer before exporting any signal.

Required behavior:

  • Hash tenant/user identifiers before export unless we explicitly choose identifiable analytics.
  • Keep trace_id, request_id, source_id, workflow_id, and ai_run_id stable enough for debugging.
  • Strip or bucket high-cardinality/private values.
  • Never log raw source material.
  • Never log raw LLM prompts/completions by default.
  • Add tests proving common sensitive fields are removed.
  • Add a kill switch: OBSERVABILITY_EXPORT_ENABLED=false must disable external export.
  • Add per-environment controls: local/dev/staging/prod.

Suggested sanitizer test cases:

phone numbers
email addresses
Vietnamese addresses
customer names
invoice IDs
bank references
raw pasted chat
raw voice transcript
raw OCR output
LLM prompt/completion body

Implementation plan

Phase 0 — decision record

Add an ADR:

docs/adr/otel-posthog-observability.md

Must state:

  • OpenTelemetry is the instrumentation standard.
  • PostHog is the first product-debug backend.
  • PostHog tracing is alpha, so Collector routing preserves optionality.
  • Raw customer content is not telemetry.
  • Infra metrics may later move to Prometheus/Grafana or equivalent.

Phase 1 — OTel foundation

  • Add OpenTelemetry SDK packages for backend runtime.
  • Configure service/resource attributes.
  • Add request ID and trace ID propagation.
  • Add structured logging with trace/span correlation.
  • Add environment variables:
OTEL_SERVICE_NAME=lumi-api
OTEL_EXPORTER_OTLP_ENDPOINT=
OTEL_EXPORTER_OTLP_HEADERS=
OTEL_TRACES_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp
OTEL_METRICS_EXPORTER=none-or-otlp
OBSERVABILITY_EXPORT_ENABLED=false
POSTHOG_PROJECT_API_KEY=
POSTHOG_OTLP_ENDPOINT=
TELEMETRY_REDACTION_STRICT=true

Phase 2 — Collector

Add local/dev Collector config:

infra/otel/collector.yaml

Pipeline requirements:

  • receive OTLP from services
  • batch export
  • retry on transient failures
  • apply attribute filtering/redaction where possible
  • export logs/traces to PostHog OTLP endpoint
  • expose its own health metrics/logs locally

Phase 3 — product-debug instrumentation

Instrument one complete golden path:

pasted chat / voice note / uploaded source
→ source parsed
→ AI extraction
→ task/risk/commitment drafted
→ user accepts/corrects/ignores
→ daily brief includes or excludes the item

This path should produce:

  • one correlated trace
  • structured logs
  • product events
  • AI run metadata
  • no raw customer content

Phase 4 — dashboards / queries

Create a minimal PostHog dashboard or saved queries for:

  • failed AI runs by workflow type
  • slowest backend routes by p95
  • worker retry/failure count
  • AI cost by day/model/workflow
  • accepted vs corrected AI outputs
  • daily brief generation failures
  • redaction drops
  • top product-visible error classes

Phase 5 — cheapest disconfirming test

Run the system for 7 days or enough realistic test traffic to answer these questions:

  1. Which tenants had failed AI tasks today?
  2. Which backend route is slowest by p95 latency?
  3. Which feature flag variant caused more errors?
  4. Can we jump from a backend error to the relevant user/session/product event?
  5. Can we debug a failed worker without SSH/log grep?
  6. Can we estimate AI cost per workflow per day?
  7. Can we detect when AI outputs are frequently corrected or ignored?
  8. Can we prove raw customer content is not exported?

Pass condition: PostHog answers at least 6/8 cleanly, with no privacy violation.

Fail condition: if p95 latency, worker debugging, alerting, or retention is weak, add a separate metrics/logs/traces backend behind the same Collector rather than replacing instrumentation.

Acceptance criteria

  • ADR exists and explains the PostHog + OTel Collector decision.
  • Backend emits OpenTelemetry traces for API requests and at least one worker job.
  • Structured logs include trace_id and request_id.
  • PostHog receives sanitized OTLP logs.
  • PostHog receives sanitized OTLP traces if enabled.
  • Product events include correlation IDs linking PostHog events to traces/logs.
  • One golden AI workflow is fully traceable end-to-end.
  • Telemetry sanitizer has tests for sensitive customer data.
  • OBSERVABILITY_EXPORT_ENABLED=false disables external telemetry export.
  • Local/dev setup works without requiring production PostHog credentials.
  • Documentation explains how to run local observability.
  • Documentation explains what must never be logged.
  • 7-day validation checklist is documented with results.

Definition of “insanely great”

Not more charts. Better judgment.

This implementation is successful when a founder/operator debugging Lumi can move from:

Customer says “Lumi invented a task / missed a promise / brief was wrong.”

to:

source input → parser → AI run → schema result → confidence → user correction → daily brief decision → trace/log/error

without grep, guessing, or exposing private customer content.

The real product advantage is not observability hygiene. It is trust repair speed.

Suggested PR sequence

  1. ADR + environment contract + telemetry naming conventions.
  2. OTel backend foundation + trace/log correlation.
  3. Collector config + PostHog export.
  4. Telemetry sanitizer + tests.
  5. Golden AI workflow instrumentation.
  6. PostHog dashboard/saved-query docs.
  7. Validation report after realistic traffic.

Implementation Audit

The repository contains the Phase 0–5 source, configuration, and focused tests described below. That is not production ingestion or validation evidence: Phase 6 remains open until the approved real-traffic checklist is executed and retained.

Phase 0 — Foundation (shipped 2026-07-07)

  • ADR: docs/ADR/039-otel-posthog-observability.md — five decisions (OTel standard, PostHog first backend, Collector optionality, no raw customer content, Prometheus optional future).
  • Config module: backend/core/observability/config.py — typed ObservabilityConfig frozen dataclass, read from env with safe defaults. Kill switch OBSERVABILITY_EXPORT_ENABLED defaults False.
  • Naming conventions: docs/observability/telemetry-naming.md — canonical span/log attributes, product event names, correlation-ID fields, AI workflow metadata, never-send list.
  • Tests: backend/tests/observability/test_config.py — 29 tests covering defaults, env overrides, header parsing, deployment environment resolution.
  • Registry: F96 Backend cell flipped 🔲→🚧 in docs/MVP_SCOPE.md.

Phase 1 — OTel SDK + trace/log correlation (shipped 2026-07-07)

  • pyproject.toml: added opentelemetry-api, opentelemetry-sdk, opentelemetry-exporter-otlp, opentelemetry-instrumentation-django to [project.dependencies].
  • backend/core/observability/tracing.py: init_observability() — gated on observability_export_enabled(), lazy-imports OTel SDK, builds Resource + TracerProvider + OTLPSpanExporter + BatchSpanProcessor. get_tracer(name) — returns real tracer when enabled, no-op _NullTracer singleton otherwise (zero OTel import in disabled path).
  • backend/core/logging.py: RequestIDFilter now injects trace_id and span_id (32/16-char hex) into log records when observability_export_enabled() — lazy import, no-op safe.
  • backend/core/apps.py: CoreConfig.ready() calls _init_observability(), wrapped in try/except.
  • .env.example: added all OTEL_, POSTHOG_, OBSERVABILITY_EXPORT_ENABLED, TELEMETRY_REDACTION_STRICT vars.
  • Tests: backend/tests/observability/test_tracing.py — disabled-path safe no-op + sys.modules assertion (no OTel import leak); enabled-path guarded with pytest.importorskip.
  • Spec audit: this file updated.
  • Registry: F96 cell stays at 🚧.

Phase 2 — Telemetry sanitizer (shipped 2026-07-07)

  • backend/core/observability/sanitizer.py: hash_identifier() (SHA-256, 16-char hex), sanitize_attributes() (hash IDs, drop sensitive keys, keep debug IDs verbatim, bucket char_count, fail-closed in strict mode), scrub_text() (redact emails/phones/long digit runs).
  • backend/tests/observability/test_sanitizer.py: 86 tests covering every spec sanitizer test case — phone numbers, emails, addresses, customer names, invoice IDs, bank refs, raw chat, voice transcript, OCR, LLM prompt/completion; debug IDs survive verbatim; input immutability; strict vs non-strict mode; char_count bucketing.
  • Exports: core/observability/__init__.py re-exports hash_identifier, sanitize_attributes, scrub_text, SENSITIVE_KEYS, DEBUG_IDS, HASH_KEYS.
  • Note: lands before export wiring (Phase 3+ Collector/PostHog export) as a privacy prerequisite. Registry cell stays 🚧.

Phase 3 — Collector config (shipped 2026-07-07)

  • infra/otel/collector.yaml — OTLP receive (gRPC + HTTP), batch processor, memory_limiter, attributes/redaction processor dropping all 19 SENSITIVE_KEYS (mirroring sanitizer.SENSITIVE_KEYS), otlphttp/posthog exporter using ${env:POSTHOG_OTLP_ENDPOINT} and ${env:POSTHOG_PROJECT_API_KEY} with retry + sending queue, debug exporter for local, health_check extension, service.pipelines for both logs and traces (traces noted as PostHog-alpha).
  • docs/observability/local-setup.md — how to run the Collector via Docker, required env vars, backend config, bold “What must NEVER be logged” list.
  • Tests: backend/tests/observability/test_collector_config.py — 20 tests: YAML parses, has receivers.otlp with gRPC+HTTP, has batch + memory_limiter + attributes/redaction, all 19 sensitive keys deleted, no extra keys, has otlphttp exporter + debug, has health_check, logs + traces pipelines wired with redaction; no hardcoded secrets (all credentials via ${env:...}).
  • Registry: F96 cell stays at 🚧.

Phase 4 — Product-debug instrumentation

  • core/observability/events.py — emit_product_event(name, props) helper: sanitizes all properties via sanitize_attributes(), logs structured JSON via lumi.product_events logger, no-op when kill-switch off, never raises.
  • 12 canonical event constants: AI_EXTRACTION_STARTED, AI_EXTRACTION_COMPLETED, AI_EXTRACTION_FAILED, ASSISTANT_RESPONSE_COMPLETED, COMMITMENT_DRAFTED, RISK_DRAFTED, TASK_DRAFTED, USER_ACCEPTED_AI_OUTPUT, USER_CORRECTED_AI_OUTPUT, USER_IGNORED_AI_OUTPUT, DAILY_BRIEF_GENERATED, UNDO_USED.
  • ASSISTANT_RESPONSE_COMPLETED is emitted after the persisted assistant message is complete and owns the user-visible response_ms/ttfa_ms latency panel; AI_EXTRACTION_COMPLETED.total_ms remains worker timing.
  • Instrumentation points: run_agent_loop_task (start/complete/fail + per-draft events), confirm_action (accepted), reject_action (corrected), undo_action (undo used), generate_brief view (brief generated).
  • Tests: test_events.py (5 unit tests) + test_golden_path_events.py (6 integration tests).

Phase 5 — Dashboards (shipped 2026-07-07)

  • docs/observability/posthog-dashboard.md: 8 PostHog insights with HogQL queries — failed AI runs by workflow type, named assistant response p95 latency, worker retry rate, AI cost proxy, accepted vs corrected ratio, daily brief generation, redaction drops, top error classes. Setup steps and escalation path (Prometheus/Loki when PostHog insufficient).
  • docs/observability/validation-checklist.md: 7-day pass/fail checklist for the 8 F96 validation questions with explicit PASS/FAIL/SKIP criteria per question.

Phase 6 — 7-day validation

  • Run against realistic production traffic with OBSERVABILITY_EXPORT_ENABLED=true.
  • Complete docs/observability/validation-checklist.md — requires ≥6/8 questions PASS.
  • On PASS: flip F96 registry from 🚧 → ✅ in docs/MVP_SCOPE.md.

References checked on 2026-07-07

  • PostHog Logs docs: OTLP/OpenTelemetry-compatible logs; no vendor-specific SDK required.
  • PostHog Distributed Tracing docs: OTLP-compatible traces/spans; currently alpha.
  • OpenTelemetry Collector docs: vendor-agnostic receive/process/export layer for traces, metrics, and logs.