Spec: F51 — Mobile Visual Testing & UX Regression Suite

Current V2 shell under test: Lumi · Work · Business · Hub.

Status: agreed

1. Problem & CEO Job

Job: Trust the signal — a broken Today screen on iPhone SE is a product failure even if nobody finds it until a CEO complains. Mobile apps span device sizes, OS versions, font scales, and RTL locales; a Vietnamese iPhone 16 Pro Max layout can overflow on an English iPhone SE or disappear in iPad split-screen.

Web has Playwright screenshots (F22) and mobile has unit tests, but neither iOS nor Android has visual regression coverage. Every layout change is therefore a blind deploy across screen sizes, string lengths, and text scaling.

This spec adds CI pixel diffs, a device matrix, accessibility/Large Text/screen-size variants, and UX smoke tests for the Rule of Two: every feature is reachable in ≤2 taps.

2. User Stories & Acceptance Criteria

iOS Visual Regression

• As an iOS developer, I want snapshot tests that catch layout regressions, so that I don’t push a broken Today screen to production.

  • SwiftUI snapshot testing via swift-snapshot-testing (Point-Free) or Apple’s XCTest with XCUIScreenshot.
  • Reference images stored in ios/LumiTests/Snapshots/.
  • CI fails if any snapshot differs from reference by >1% pixel diff.
  • Snapshot naming: <ScreenName>_<device>_<locale>_<textSize>.png.

• As a QA engineer, I want screenshots captured on multiple device sizes, so that iPhone SE, iPhone 16 Pro, and iPad are all verified.

  • Device matrix: iPhone SE (375×812), iPhone 16 Pro (393×852), iPhone 16 Pro Max (430×932), iPad 11” (820×1180).
  • Localized snapshots: en and vi per screen (string-length divergence catches overflow).
  • Dynamic Type variants: standard, Large, Accessibility ExtraLarge.

Android Visual Regression

• As an Android developer, I want screenshot tests with Roborazzi, so that the same regression detection works on both platforms.
  • Roborazzi (already set up per android/README.md screenshot test infrastructure — repurpose for Lumi UI).
  • Reference images stored per module.
  • CI runs verifyRoborazziDemoDebug (or renamed Lumi variant).
  • Device matrix: Pixel 7 (412×915), Pixel 7 Pro (412×960), Pixel Tablet (800×1280).

UX Smoke Tests (Rule of Two)

• As a product owner, I want automated tests that verify every feature is reachable in ≤2 taps, so that navigation depth doesn’t drift.
  • iOS XCUITest: tap Lumi tab → verify assistant home loaded (1 tap). Tap Work tab → verify execution home loaded (1 tap). Tap Hub → tap Tasks → verify full-directory reach in 2 taps.
  • Android Compose UI test: same navigation assertions.
  • Assertion: no feature requires >2 taps from any of the 4 bottom tabs.
  • CI fails if any feature needs 3+ taps.

Accessibility & Large Text

• As a user with vision needs, I want the app to work at all Dynamic Type sizes, so that text scaling doesn’t break layouts.
  • iOS: snapshot each screen at default, large, ax1, ax3 sizes.
  • Android: snapshot each screen at default, 1.3x, 1.5x, 2.0x font scale.
  • No text clipping. No button truncation. Minimum touch target 48×48dp preserved.

Pull Request Gates

• As a mobile team lead, I want visual testing to run on every PR, so that regressions are caught before review.
  • iOS PRs: guarded, serial xcodebuild test includes snapshot tests. CI uploads diff images as artifacts.
  • Android PRs: ./gradlew verifyRoborazziDemoDebug runs. CI uploads comparison images.
  • Failed snapshots: CI comment on PR with side-by-side diff images (reference vs actual vs diff).
  • UX smoke tests run in same CI workflow. Both must pass before merge.

3. Scope Boundaries

• In:
• iOS: snapshot testing framework setup, reference images for 4 key shell screens (Lumi, Work, Business, Hub), 4 device sizes, 2 locales, 3 text sizes. Pulse and Company Hub are nested surfaces, not additional shell screens.
• Android: Roborazzi configuration, reference images same matrix.
• UX smoke tests: automated navigation assertions (Rule of Two).
• CI integration for both platforms.
• PR diff comments for visual failures.
• Out:
  • Android performance benchmarking — covered by F102 (Macrobenchmark + Baseline Profile + CI regression gate), not by visual testing.
  • Snapshotting every screen in the app — 4 key shell screens cover the primary paths. Pulse and Company Hub remain nested surfaces; edge-case screens (Settings, HQ) are manual review.
  • Snapshotting every locale (62 locales) — only en + vi (the 2 active development locales).
  • Pixel-perfect comp diffing for animations — only static screens after load.
  • Performance benchmarking (separate from visual regression).
  • E2E flow testing (covered by existing Playwright suite — mobile visual testing is layout, not interaction).
  • Manual accessibility audit — automated snapshot + text scaling only.

4. Data Model

No database changes. Configuration only.

4.1 Snapshot Reference Storage

iOS (ios/LumiTests/Snapshots/):

Snapshots/
  LumiScreen_iphoneSE_en_default.png
  LumiScreen_iphoneSE_vi_default.png
  LumiScreen_iphoneSE_en_large.png
  LumiScreen_iphone16Pro_en_default.png
  LumiScreen_iphone16Pro_vi_default.png
  LumiScreen_iphone16ProMax_en_default.png
  LumiScreen_ipad11_en_default.png
  WorkScreen_iphoneSE_en_default.png
  ... (4 screens × 4 devices × 2 locales × 3 text sizes = 96 reference images)

Android (android/app/src/testDemo/snapshots/):

src/testDemo/snapshots/
  LumiScreen_pixel7_en_default.png
  LumiScreen_pixel7_vi_default.png
  ... (same matrix)

4.2 CI Artifacts

ci-artifacts/
  visual-diffs/
    PR_<number>/
      LumiScreen_iphoneSE_vi_default_reference.png
      LumiScreen_iphoneSE_vi_default_actual.png
      LumiScreen_iphoneSE_vi_default_diff.png
      index.html  (side-by-side gallery for review)

5. API Contract

No API changes.

6. UI / UX

No application UI changes. Developer workflow change:

1. Developer changes a layout in the Lumi shell screen.
2. CI snapshot test fails: LumiScreen_iphoneSE_vi_default.png differs by 3.2%.
3. PR comment shows side-by-side: reference (correct) | actual (overflowed text) | diff (red highlight).
4. Developer fixes the layout or updates the reference with record mode.
5. PR re-runs. Snapshot passes. Merge unblocked.

6.1 Snapshot Update Flow

# iOS: record references on one approved existing simulator UDID; never use a name or clone
IOS_UDID=<approved-udid>
scripts/wait-for-resources.sh --min-mb 8192 --label 'iOS snapshot recording' -- \
  xcodebuild test -project ios/Lumi.xcodeproj -scheme Lumi \
  -destination "platform=iOS Simulator,id=$IOS_UDID" \
  -parallel-testing-enabled NO \
  -parallel-testing-worker-count 1 \
  -maximum-concurrent-test-simulator-destinations 1 \
  -only-testing:LumiTests/SnapshotTests/recordSnapshots

# Android: record new references
./gradlew recordRoborazziDemoDebug

7. Test Plan

8. Current Behavior

• No iOS snapshot/visual regression testing.
• Android has Roborazzi infrastructure (from NIA template) but no Lumi-specific reference images or tests; it does have focused JVM navigation proof and instrumented F79 shell/Rule-of-Two coverage.
• No complete cross-platform UX smoke suite: Android covers the shell path, while iOS XCUITest coverage and the wider route matrix remain open.
• No CI visual diff pipeline.
• E2E testing is web-only (Playwright in frontend/tests/e2e/).
• Existing mobile proof is split across iOS UnitTests/, Android JVM tests, and Android instrumented tests; none of it is yet a visual-regression baseline or full named-device parity proof.
• Navigation contract note (2026-08-05): route the shell smoke cases through F79, not F30’s older tab naming. Mobile visual testing must verify the four shell destinations Lumi · Work · Business · Hub; Pulse and Company Hub are nested surfaces. Today, Brief, Ask Lumi, and Notifications remain supporting modes.

9. Error States

10. Diagrams & Docs Touched

• F30 mobile verification docs and simulator/browser workflow docs
• Platform testing diagrams if capture becomes a standard release gate

11. Implementation Audit

Edge Cases Catalog (formerly §9, 2026-06-14; 8 items): baseline filename matrix encodes device×locale×text-size, deterministic fixtures via setSnapshotMode(_:), vi overflow must not be masked by 1.5% threshold, CI simulator references only (no physical device baselines), Roborazzi vs swift-snapshot 1.5% aligned qualitatively, OS upgrade triggers mass-diff report with version label, UX smoke adaptive 30s floor timeout, advisory ax3 failures don’t block non-a11y PRs. Full catalog in git history.

12. Open Questions / Decisions

12.1 Contract Graph reconciliation (2026-07-15)

The Contract Graph records the current implementation boundary rather than the target-state checklist. The source-level Android/iOS tab checker, Android Rule-of-Two instrumentation flow, iOS Rule-of-Two XCTest declarations, and Roborazzi helper are real repository surfaces and are registered as F51 proofs. The agreed snapshot matrix, iOS snapshot suite, named-device execution, CI visual-diff jobs, artifact uploads, and PR comparison comments are not present in this checkout and remain explicit gaps. No browser, native runtime, live-provider, Sentry, Redis, or production evidence is inferred from source or test collection.

1. Snapshot testing library for iOS: swift-snapshot-testing (Point-Free) vs Apple’s XCTest built-in screenshot support. Decision: swift-snapshot-testing — more mature, better diff output, SwiftUI-native, and handles device-size simulation without running different simulators.
2. Reference image storage: Git LFS or regular git? Decision: regular git. 96 PNGs × ~50KB each = ~5MB total. Small enough for standard git. Git LFS adds CI complexity for marginal benefit at this scale.
3. CI simulator matrix: Run on all 4 iOS devices or sample 2? Decision: run all 4. Xcode Cloud supports parallel simulators. On GitHub Actions, use matrix strategy with 4 jobs. Snapshot tests are fast (~30s per device).
4. Pixel diff threshold: 1% tolerance is sufficient for most layouts. But antialiasing differences between CI and local machines can cause false positives. Decision: start at 1.5% tolerance. Tighten to 1% after a month of no false positives.

13. Bug / Change Log

2026-06-18 — Align mobile visual smoke contract with F79 shell

Type: Documentation Change

Summary:

• Replaced the old shell-smoke example (Today / Brief / Workspace) with the new F79 shell language.
• Added an explicit note that mobile visual testing must verify the five operating tabs Pulse · Work Graph · Business · Company Hub · Workspace, and also verify that Today, Brief, Ask Lumi, and Notifications are nested modes rather than permanent tabs.

Reason:

• Leaving the old smoke example in place would cause visual-regression work to preserve the superseded tab model.

Impact:

• Future F51 implementation and review criteria now align with the mobile IA spec set.

Tests:

• None. Spec-only change.

2026-06-14 — Spec restructured to TEMPLATE.md canonical order (§1–§13)

Type: Documentation Change

Summary:

• §9 Edge Cases (8 items) absorbed into §11 Implementation Audit note.
• §12 Implementation Audit converted from “Not built” narrative + Implementation Order to canonical 4-column table (7 rows, all ⚠️ gaps — F51 not yet built).
• Sections renumbered: §10 ES → §9, §11 Diags → §10, §12 Audit → §11, §13 OQ → §12, §14 BL → §13.

Reason:

• Align with TEMPLATE.md canonical section order per SP fleet restructuring pass.

Impact:

• Spec authors and future implementation agents.

2026-06-13 — Spec hardening: numbered edge cases, detailed error matrix, status → agreed

Type: Documentation Change

Summary:

• Replaced §9 loose bullet points with 8 numbered edge cases (9.1–9.8) covering baseline matrix encoding, deterministic fixtures, localization overflow tolerance, simulator vs physical device differences, cross-platform threshold alignment, OS version upgrade regression, UX smoke test timeouts on slow CI, and advisory vs critical snapshot failure distinction.
• Replaced §10 two-sentence prose with 10-row detailed error matrix (user-visible/internal/log/retry).
• Updated spec status from draft to agreed.

Reason:

• Spec needed hardening per TEMPLATE.md. Edge cases were loose prose, error states lacked structured detail.

Impact:

• Spec authors and future implementation agents.

Tests:

• Spec-only change — no code tests affected.