shipped (merged in F56)

Spec: F55 — Upgrade from Gunicorn to Granian

Status: shipped (merged in F56)

1. Problem & CEO Job

Job: Decide fast. The current WSGI+gunicorn+gthread stack (2 workers × 4 threads = 8 concurrent slots) was a pragmatic choice for v1 (see ADR-008), but it’s now a bottleneck. Each SSE connection, Celery task callback, or long-running AI extraction consumes one of 8 slots — under load, the backend saturates and all requests queue. Granian is an ASGI/RSGI server written in Rust that eliminates the GIL bottleneck, supports true async concurrency (thousands of concurrent connections per process), native HTTP/2, and uses a single-process multi-thread model with zero shared-state contention. For the CEO, this means the app stays responsive during peak hours, AI chat flows don’t block other requests, and the team ships no new infrastructure to support it.

2. User Stories & Acceptance Criteria

  • As a CEO, I want the app to stay fast during busy hours so that I never wait for a page to load while someone else’s AI extraction is running.
    • Given 20+ concurrent SSE connections and 5 simultaneous AI extractions, when all are active, then API response times for non-SSE requests stay under 500ms.
  • As an admin, I want zero-downtime deploys so that no team member’s work is interrupted.
    • Given a rolling deploy with the new server, when tasks cycle, then existing SSE streams reconnect without data loss.
  • As a developer, I want hot-reload during local development so that I iterate faster without restarting containers.
    • Given a source code change, when --reload is active, then the server restarts automatically.

3. Scope Boundaries

  • In: Replace gunicorn with Granian in production Docker image; update entrypoint.sh, pyproject.toml (and delete requirements.txt — F56 confirmed deletion as the right action, not an in-place update), deploy docs, diagrams.
  • Out: Switching from WSGI to ASGI in the Django application layer — this spec preserves config.wsgi:application via Granian’s WSGI compatibility layer (--interface wsgi). Full ASGI migration of Django views/apps is explicitly future work.

4. Data Model

No models touched or added.

5. API Contract

No API changes. The server bind, port, and health check remain identical (0.0.0.0:8000, /api/health/).

6. UI / UX

No user-facing changes. The app behaves identically — faster under load, same routes, same UI.

7. Test Plan

#CaseTypeWhere
1Health endpoint returns 200e2ePOST /api/health/
2SSE streams survive rolling deploye2eManual: scale ECS, watch stream reconnect
3Granian binary resolves in Docker buildinfradocker build . succeeds
4Concurrency under loadperfNot automated in v1

No backend test changes needed — Granian is a process-level replacement, not an app-level change. Existing pytest suite must pass unchanged.

8. Current Behavior

The backend previously ran on gunicorn (WSGI, v22.0–26.0) with:

gunicorn config.wsgi:application \
    --bind 0.0.0.0:8000 \
    --workers 2 \
    --worker-class gthread \
    --threads 4 \
    --timeout 120
  • 2 worker processes × 4 threads = 8 concurrent request slots
  • WSGI interface (sync), gthread worker class for SSE compatibility

Current (as of F56): Granian ASGI — see F56-python314-uv-asgi.md.

9. Error States

  1. Startup failure. Granian fails to start due to Python import error, missing binding, or invalid flag. Result: the Docker container exits with code 1, ECS health check fails, the task is replaced. No user impact beyond the rolling deploy cycle. Mitigation: pin Granian version in pyproject.toml, test the entrypoint locally with docker compose up before deploying.

  2. Worker crash. A Granian worker thread panics or exits due to unhandled exception in application code. Granian’s worker model isolates threads — a single thread crash does not take down the entire process; other threads continue serving. Mitigation: keep --backlog at Granian default (typically 2048); monitor granian_worker_crashes_total via the metrics endpoint if enabled.

  3. Out of memory (OOM). The container’s memory limit (1024 MB per current ECS task definition) is exceeded. The kernel OOM killer terminates the Granian process; ECS restarts the container. Mitigation: monitor memory in CloudWatch (MemoryReservation metric), consider raising the ECS task memory allocation if SSE + AI extraction load pushes past 80%.

  4. Port binding failure. The configured --bind 0.0.0.0:8000 fails because the port is already in use or the container lacks permissions. Granian exits immediately with a clear error message. On local dev: verify no orphan containers occupy port 8000 before docker compose up; use exec in the entrypoint so the shell does not hold the port open.

  5. --reload file-watch limits. The inotify watcher has a default max_user_watches of 8192 on Linux. The Django project tree may exceed this under git operations, causing silent reload misses. Mitigation: in Docker dev containers, sysctl -w fs.inotify.max_user_watches=524288; log a warning in the entrypoint if --reload is active and the watch limit is low.

10. Diagrams & Docs Touched

DocumentChange
deploy/AGENTS.mdReplace gunicorn references with Granian, update rationale
deploy/DIAGRAMS.mdUpdate architecture labels
DIAGRAMS.mdUpdate backend subsystem label
backend/entrypoint.shReplace gunicorn with Granian
backend/pyproject.tomlReplace gunicorn>=22.0,<27.0 with granian>=1.6,<2.0
backend/requirements.txtDeleted in F56 (not updated — pyproject.toml became the sole dependency manifest; the planned in-place replacement was superseded by F56’s deletion decision)
docs/ADR/008-messaging-app-and-sse.mdNote ASGI migration now feasible

11. Implementation Audit (2026-06-14)

#ClaimStatusEvidence
1backend/entrypoint.sh uses Granian, not Gunicorn✅ matchMerged in F56; backend/entrypoint.sh updated
2backend/pyproject.toml has granian>=1.6,<2.0, not gunicorn✅ matchMerged in F56
3backend/requirements.txt deleted (not updated)✅ matchMerged in F56 — F56 deleted the file rather than updating it; pyproject.toml is the sole dependency manifest
4--interface wsgi (not asgi) for backwards compatibility✅ match§12 open decision: start with wsgi — confirmed
5Existing pytest suite passes unchanged (no app-level WSGI→ASGI code changes)✅ matchSpec §3: Granian is process-level replacement only
6Health check endpoint returns 200 on Granian✅ matchE2E always hits /api/health/ first
7Concurrency performance target (< 500ms under 20 SSE + 5 extractions)⚠️ unverified§7 test case 4: “Not automated in v1” — load test not yet run
8SSE streams reconnect after rolling deploy⚠️ unverified§7 test case 2: “Manual” only — no automated proof

Gaps from Edge Cases (formerly §6, now integrated):

  • Port conflicts on startup (§9 error 4): use exec in entrypoint; ECS lifecycle prevents collisions during normal rolling deploys but not manual restarts.
  • Reload edge cases on macOS/Docker Desktop: inotify bridge latency 1–3s; acceptable for local dev only (--reload not used in production).
  • Rolling deploy SSE reconnect: 5–15s window; frontend SSE client implements exponential backoff reconnect; no data loss because events persist to DB.
  • WSGI→ASGI migration path: when views are async-ready, --interface asgi + config.asgi:application is a one-flag change. Tracked in §12.

12. Open Questions / Decisions

  • Interface: Start with --interface wsgi for zero application change, or --interface asgi to unlock async views now? Decision: --interface wsgi for strict backwards compatibility; switch to ASGI in a follow-up after verifying all views work.
  • Process model: --processes 2 --threads 8 (16 concurrent slots) or --processes 1 --threads 16 for simpler state? Decision: --processes 2 --threads 8 to match existing 2-worker topology; tune after load testing.
  • HTTP version: Enable HTTP/2 from start? Decision: yes, --http2 — no client-side changes needed (ALB can negotiate).
  • Seccomp / capabilities: Granian uses seccomp(2) for sandboxing. Verify in dev that this doesn’t conflict with the Docker container’s seccomp profile before deploying.

13. Bug / Change Log

2026-07-02 — Fix stale paths: requirements.txt “updated” → “deleted” (batch 3)

Type: Documentation

Summary:

  • F56 deleted backend/requirements.txt rather than updating it (F56 locked pyproject.toml as the sole manifest).
  • §3 Scope Boundaries: noted that deletion was the actual action, not an in-place update.
  • §10 Diagrams table row: corrected “Replace …” to “Deleted”.
  • §11 row 3: corrected “updated” to “deleted” with the correct explanation.
  • No code changes (spec is archived).

2026-07-02 — Spec drift review (batch10)

Type: Documentation Change

Summary:

  • Reviewed §2 ACs against §11 audit evidence. All three ACs remain [ ]:
    • AC 1 (< 500ms under 20 SSE + 5 extractions): §11 row 7 is ⚠️ unverified — load test not yet run.
    • AC 2 (SSE reconnect on rolling deploy): §11 row 8 is ⚠️ unverified — manual check only.
    • AC 3 (hot-reload on source change): no §11 row covers --reload behavior.
  • No ACs changed. No code changes.

2026-06-14 — Restructure to TEMPLATE.md section order (SP fleet)

Type: Documentation Change

Summary:

  • Renumbered sections to match TEMPLATE.md (§6 UI/UX, §7 Test Plan, §8 Current Behavior, §9 Error States, §10 Diagrams, §11 Implementation Audit, §12 Open Questions, §13 Bug Log).
  • Moved Edge Cases content (formerly §6, inserted by D-r20) into §11 audit gaps.
  • Expanded §11 from 1-sentence verdict to 8-row audit table.

Reason:

  • D-r20 had inserted Edge Cases at §6 and renumbered everything after — non-standard vs TEMPLATE.md.

Tests:

  • No code changes.

2026-06-12 — Harden spec: add Edge Cases section, enhance Error States section (D-r20)

Type: Documentation Change

Summary:

  • Added Edge Cases with 5 numbered items (port conflicts, reload edge cases, rolling deploys, SSE reconnection, WSGI→ASGI migration path).
  • Enhanced Error States with 6 numbered items.

Impact: doc-only change.

2026-07-15 — Contract Graph supersession boundary

The current entrypoint and archived status are now covered by the small F55 boundary contract and hermetic unittest proof. F56 owns the active Python 3.14/uv/Granian ASGI implementation; F55’s load-performance, rolling-deploy SSE, and hot-reload checks remain unverified operational gaps.