Spec: F34 — User Uploads (Pre-signed S3 POST)

Status: ✅ shipped (backend 2026-06-11; frontend + E2E 2026-06-12)

1. Problem & CEO Job

CEO job: Trust the signal — every uploaded chat image, pulse attachment, or task-evidence file must be consistently stored, CDN-served, and traceable to its source.

Without this feature, chat media, pulse photos, approval attachments, and Hub images would each invent an upload mechanism, creating inconsistency, security gaps, and storage chaos. One pre-signed endpoint gives every file the same path convention, security model, CDN delivery, and audit trail: no rework or parallel systems.

2. User Stories & Acceptance Criteria

• As a member, I want to upload a file, so that I can attach evidence, images, or documents anywhere in Lumi.

  • Given valid filename, content_type, and byte_size, when I POST to /api/uploads/presign/, then I receive a pre-signed POST URL + S3 form fields + upload_id + public CDN URL
  • Given a completed upload, when I call POST /api/uploads/{id}/complete/, then the upload is marked as completed and an EvidenceSource is created

• As any role, I want the upload endpoint to enforce security, so that no one can abuse it.

  • Given a viewer role, when I call the presign endpoint, then I receive 403
  • Given an oversized file request, when I call the presign endpoint, then I receive 400 with a clear error
  • Given a disallowed MIME type, when I call the presign endpoint, then I receive 400
  • Given more than 30 requests per minute, when I call the presign endpoint, then I receive 429

3. Scope Boundaries

• In: Pre-signed POST URL generation, Upload tracking model, file size+MIME validation, S3 key convention, CDN public URL generation, EvidenceSource linkage, complete/ endpoint
• Out: Resumable/multipart uploads (future), image processing/resizing (future, use CDN image optimization), virus scanning (future hook), file expunging/deletion API (future), pre-signed GET URLs for private files (future — all v1 files are public-read via CDN)

4. Data Model

Upload

class Upload(UUIDModel):
    STATUS_CHOICES = [
        ("pending", "Pending"),
        ("completed", "Completed"),
        ("expired", "Expired"),
    ]

    organization = FK("organizations.Organization", CASCADE, related_name="uploads")
    user = FK("organizations.OrganizationMember", SET_NULL, null=True, related_name="uploads")
    filename = CharField(max_length=1024)
    content_type = CharField(max_length=255)
    byte_size = PositiveBigIntegerField()
    s3_key = CharField(max_length=2048, unique=True)
    status = CharField(max_length=20, choices=STATUS_CHOICES, default="pending")
    expires_at = DateTimeField()
    public_url = CharField(max_length=2048, blank=True)

    class Meta:
        db_table = "uploads"
        ordering = ["-created_at"]
        indexes = [
            Index(fields=["organization", "-created_at"]),
            Index(fields=["organization", "status"]),
            Index(fields=["expires_at"]),
        ]

EvidenceSource Linkage

On complete/, an EvidenceSource is created with source_type="uploaded_file" and linked to the Upload. Calling models (Task, Commitment, Pulse, etc.) can reference the EvidenceSource via their existing evidence_sources M2M.

5. API Contract

POST /api/uploads/presign/

POST /api/uploads/{id}/complete/

Tenant scoping: all queries filter by organization_id=self.request.organization_id. S3 keys are generated server-side, never from user input.

6. UI / UX

Upload is triggered by consumers, not a standalone upload page. FileAttachButton (Paperclip icon, hidden <input type="file">, shadcn Button) is the shared entry point wired into ChatPage as the first consumer. No dedicated upload route — the useUpload hook (presign → S3 POST → complete) is the integration surface for all future consumers.

• i18n: uploads namespace (6 keys).
• States: uploading spinner, error toast, attachment pill with dismiss.

7. Test Plan

100% backend coverage required.

8. Current Behavior

The full F34 contract is shipped (backend + frontend + E2E):

• Backend: apps/uploads — Upload model, /api/uploads/presign/, /api/uploads/{id}/complete/, /api/uploads/, S3 key generation, CDN URL generation, EvidenceSource(source_type="uploaded_file") linkage.
• Frontend: UploadPresignResponse + UploadRecord types; uploadsApi.presign() + uploadsApi.complete() in endpoints.ts; useUpload hook (presign → S3 POST → complete, returns publicUrl); FileAttachButton component (Paperclip icon, MIME allowlist, 10 MB client validation); wired into ChatPage as the first consumer (attachment pill + URL appended to message on send).
• Locale: uploads namespace in all 8 supported language dirs (en, en-AU, en-CA, en-GB, en-IN, en-SG, en-ZA, vi) with 6 keys each.
• E2E: frontend/tests/e2e/uploads.spec.ts — 3 tests: attach button visible, upload flow with mocked S3/presign, oversized-file client rejection.

9. Error States

10. Diagrams & Docs Touched

• backend/DIAGRAMS.md — add Upload model to ER diagram, state machine
• docs/MVP_SCOPE.md — add F34 entry
• backend/AGENTS.md — add uploads app responsibilities
• docs/devops/aws.md — document upload path convention, S3 lifecycle policy notes
• docs/specs/F29-internal-chat.md — note that chat media uses this endpoint

11. Implementation Audit (2026-07-14)

The canonical Contract Graph declaration is contracts/features/F34.cue. The current code-vs-doc audit is F34-user-uploads-audit.md. The API-focused contract owns all 10 upload surfaces and the collected backend/security proof; live S3/CDN and browser/native runtime evidence remain explicit gaps.

Edge Cases Catalog (formerly §10, 2026-06-14; 4 items): idempotent finalize (no double attachments on retry), membership-revoked presign finalize rejected, plan-limit failure preserves reason, MIME content-validation mismatch denied. Full catalog in git history.

12. Open Questions / Decisions

• Pre-signed POST vs PUT: POST chosen for browser form compatibility. PUT can be added later for mobile/native clients.
• Complete trust model: Client self-reports completion. Orphan pending uploads cleaned by S3 lifecycle policy (delete after 7 days).
• Max file size: 10 MB default, overridable via UPLOAD_MAX_SIZE env var. Per-type overrides deferred.

13. Bug / Change Log

2026-07-02 — Correct stale page path in §11 row 8 (batch-1 spec audit)

Type: Documentation Fix

Summary:

• §11 row 8: Updated frontend/src/pages/ChatPage.tsx → frontend/src/pages/connect/ChatPage.tsx to reflect the IA restructure that moved pages into subdirectories. Verified by find frontend/src -name ChatPage.tsx = frontend/src/pages/connect/ChatPage.tsx.

2026-07-02 — Spec drift fix: check shipped ACs against §11 audit

Type: Documentation Truth Fix

Summary:

• Checked all 6 previously open [ ] ACs in §2 that §11 confirms as implemented:
  • AC1 (presign → URL + S3 fields + upload_id + CDN URL) — rows 2, 4 (/api/uploads/presign/ ✅ + test case 1 “Presign returns URL + fields for valid request”)
  • AC2 (complete → completed + EvidenceSource) — row 3 (EvidenceSource(source_type=“uploaded_file”) linkage on complete ✅)
  • AC3 (viewer → 403) — row 4 (100% coverage; test case 5 “Presign rejects viewer role (403)”)
  • AC4 (oversized file → 400) — row 4 (test case 3 “Presign rejects oversized file (400)”)
  • AC5 (disallowed MIME → 400) — row 4 (test case 2 “Presign rejects invalid MIME type (400)”)
  • AC6 (>30 req/min → 429) — row 4 (test case 6 “Presign rate-limited at 30/min (429)“)

2026-07-14 — Contract Graph ownership and proof alignment

Type: Documentation and proof-traceability change

Summary:

• Added the canonical contracts/features/F34.cue contract for all 10 upload API surfaces, presign validation, completion/evidence linkage, tenant boundaries, and shared persistence.
• Attached stable proof IDs to existing upload endpoint and multimodal isolation tests.
• Kept F87’s processing behavior, live S3 delivery, and browser/native runtime evidence explicit rather than inferring them from mocked storage or source presence.

2026-06-15 — Fix: corrupted S3 credential broke browser uploads

Type: Bug Fix

Summary:

• Root cause: the MEDIA_S3_ACCESS_KEY_ID secret was wired with a stray echo -n artifact, so its value was literally "-n AKIA…". boto3 signed the presigned POST policy with that polluted key, and every browser upload to S3 then failed with an opaque CORS Missing Allow Origin / signature error. The presign step itself succeeded, masking the cause.
• Fix (root): new dependency-free core/secrets.py::clean_secret() strips surrounding whitespace and stray leading shell-flag tokens (-n , -e , -E ). Applied in config/settings/production.py to MEDIA_S3_ACCESS_KEY_ID, MEDIA_S3_SECRET_ACCESS_KEY, bucket, and region as they enter the S3 storage config — so boto3 signs with a clean key. Sanitizing the field after signing would not work; the signature is computed from the key.
• Fix (defense): apps/uploads/services.py::_assert_valid_presigned_fields() validates the generated AWSAccessKeyId against the AWS key format and raises a clear, logged RuntimeError at presign time if it is malformed, turning a silent client-side CORS death into a diagnosable server error. Skips SigV4 fields (x-amz-credential, no AWSAccessKeyId).

Tests:

• backend/tests/test_secrets.py — 9 tests for clean_secret (pass-through, None/empty, whitespace, the exact -n AKIA… bug, stacked tokens, no mid-value stripping).
• backend/tests/test_uploads.py — +3 tests: valid key passes, malformed key raises, SigV4 fields allowed.
• frontend/src/test/AccountPage.avatar.test.tsx — 6 tests covering the /account avatar upload (full presign → S3 POST → complete → updateProfile flow, verbatim field forwarding, S3/CORS failure → error toast, presign failure, cancel).
• 100% coverage on core/secrets.py and apps/uploads/services.py.

Impact:

• Browser avatar/file uploads succeed once the secret is clean; misconfigured credentials now fail loudly and early rather than silently in the browser.

2026-06-14 — Spec further restructured: §6 S3 Key Convention + §7 Security → appendix; §6 UI/UX added

Type: Documentation Change

Summary:

• §6 S3 Key Convention → §14 Appendix; §7 Security → §15 Appendix.
• Added minimal §6 UI/UX (FileAttachButton + useUpload hook model).
• §8 Test Plan → §7, §9 Current Behavior → §8, §10 Error States → §9, §11 Diagrams → §10, §12 Audit → §11, §13 OQ → §12, §14 BL → §13.

Reason:

• Bring §6–§7 non-standard sections into canonical order per TEMPLATE.md.

2026-06-14 — Spec restructured to TEMPLATE.md canonical order

Type: Documentation Change

Summary:

• §10 Edge Cases (4 items) absorbed into §12 Implementation Audit note.
• §14 Implementation Audit moved from after §13 OQ to §12 (before OQ), converted from 2-col table to canonical 4-column table (10 rows, all ✅).
• Sections renumbered: §11 ES → §10, §12 Diags → §11, Audit → §12, §13 OQ stays §13, §15 BL → §14.

Reason:

• Align with TEMPLATE.md canonical section sequence per SP fleet restructuring pass.

Impact:

• Spec authors and future implementation agents.

2026-06-12 — Frontend + E2E shipped (P17)

Type: Feature

Summary:

• useUpload hook: presign → S3 FormData POST → complete; returns publicUrl, exposes uploading + error + reset.
• FileAttachButton component: shadcn Button + Paperclip icon, hidden <input type="file">, MIME allowlist + 10 MB client guard.
• ChatPage wired as first consumer: file picker → upload → attachment pill with dismiss; public URL appended to message on send.
• uploads locale namespace (6 keys) added to all 8 supported language dirs.
• frontend/tests/e2e/uploads.spec.ts: 3 Playwright tests (button visible, upload flow with mocked S3, oversized-file client rejection).

Reason:

• Completes the backend-only partial ship; F34 is now fully usable end-to-end.

Tests:

• 814 frontend unit tests pass, 0 TS errors.
• E2E upload spec added (run with make e2e).

2026-06-12 — Backend-ship doc sync

Type: Doc Fix

Summary:

• Updated F34 from greenfield/draft wording to the actual backend-shipped, frontend-pending state.

Reason:

• The repo already contains the upload backend and its test suite, but the spec still claimed the feature did not exist.

Impact:

• The spec now matches the registry and current implementation boundary.

Tests:

• No code changes. Verification comes from backend/tests/test_uploads.py.

Notes:

• A future implementation plan is still needed for the frontend/E2E half.

14. S3 Key Convention (Appendix)

user_upload/{member_uuid}/{YYYY-MM-DD}/{unix_timestamp}-{8_hex_chars}.{extension}

Example: user_upload/0198ac45-.../2026-06-11/1718100000-a3f2b9c1.pdf

15. Security Model (Appendix)