iPadCast — Native iPad Integration Spec

This document is the contract the native iPad sender must follow. It is written from the shipped code: src/lib/signaling.ts, src/lib/sender.ts (reference sender), src/lib/receiver.ts, the /api/public/* routes and the database row-level security (RLS) rules. If the code and this page disagree, the code wins — please report it.

1. Overview

┌──────────── iPad ─────────────────────────────┐
│  iPadCast app (SwiftUI)                       │
│   pairing UI · anonymous sign-in · approval   │
│        │  App Group + Keychain access group   │
│        ▼  (sessionId, access/refresh token)   │
│  Broadcast Upload Extension (ReplayKit)       │
│   RPBroadcastSampleHandler → WebRTC sender    │
└────────┬──────────────────────────────┬───────┘
         │ HTTPS JSON /api/public/*      │ WebRTC media (DTLS-SRTP)
         │ Supabase REST + Realtime      │ H.264 video (+ audio)
         ▼                               ▼
┌── Backend ─────────────┐     ┌── Windows PC ───────────────┐
│ casting_sessions       │◄───►│ Chrome receiver (this app)  │
│ signaling_messages     │     │ /dashboard → /viewer/:id    │
└────────────────────────┘     └─────────────────────────────┘

• Session state (create, pair, approve, connected, heartbeat, end) changes only through the /api/public/* HTTPS JSON endpoints. Clients can never write casting_sessions directly.
• Signaling (offer, answer, ICE candidates, disconnect) uses rows in signaling_messages. Clients insert them through the Supabase REST API and receive them through Supabase Realtime, with RLS deciding who can see what.
• Media flows peer-to-peer over WebRTC from the iPad (offerer, send-only) to Chrome (answerer, receive-only).

2. Authentication

The publishable key is public by design. Every permission is enforced by RLS and by the endpoints, never by keeping the key secret.

1. Sign in anonymously with supabase-swift: try await supabase.auth.signInAnonymously(). This gives a real user id (auth.uid()), and the iPad is the sender for any session it pairs with.
2. Persist the session; supabase-swift refreshes the access token automatically (autoRefreshToken). Before each API call, read supabase.auth.session.accessToken (this refreshes it if expired).
3. Call every /api/public/* endpoint with:

   Authorization: Bearer <supabase access token>
   Content-Type: application/json
   

   The server validates the JWT and derives the caller's uid from it; any uid in a body is ignored.
4. Errors always have this shape:

   { "error": { "code": "invalid_code", "message": "Invalid or expired pairing code" } }
   

   Common codes: 401 unauthorized (missing or invalid token), 500 server_error.

Use one anonymous identity per installation and keep it stable. A new identity per attempt does not bypass the per-IP pairing limit.

3. QR pairing

The receiver's QR code encodes:

https://sideview.svry.co/pair?t=<token>&s=<sessionId>

• t: 32 random bytes, base64url-encoded (no padding). It is single-use and only its SHA-256 hash is stored.
• s: the session UUID.

Parse it with URLComponents → queryItems named t and s; reject it if either is missing or s isn't a UUID. Then call session-pair with { "token": t, "sessionId": s, "deviceName": … }.

Universal Links: not configured yet. Today /pair on the web only shows "Open this in the iPadCast iPad app". When Universal Links are added (an apple-app-site-association file on https://sideview.svry.co covering /pair), the app will receive the same URL in onContinueUserActivity and must parse it exactly as above. Until then, scan the code in-app (AVFoundation / VisionKit DataScannerViewController).

4. PIN pairing

There are six digits. The receiver shows them grouped (e.g. 482 913), and any non-digit characters are stripped server-side. Only an HMAC-SHA256 of the PIN (keyed with a server secret) is stored, and the PIN is single-use.

POST /api/public/session-pair

Request (PIN or QR):

{ "pin": "482913", "deviceName": "Jane's iPad Pro" }

{ "token": "<t>", "sessionId": "<s>", "deviceName": "Jane's iPad Pro" }

deviceName is optional (default "iPad", max 80 characters).

Success 200:

{ "sessionId": "7f0c…", "status": "pairing" }

Brute-force budget (enforced server-side):

• 5 failures per IP and per user per rolling 10 minutes.
• Each attempt is recorded before the check, so parallel bursts cannot get past it.
• Five wrong QR tokens for one session expire that session, so the receiver must generate a new code.
• The pairing window is 5 minutes (expires_at).
• The IP comes from Cloudflare's cf-connecting-ip on the published host. In local or preview environments it falls back to x-forwarded-for, which a client can spoof; the per-user limit still applies there.

On success the server:

• sets the session's sender_user_id to you and status to pairing;
• deletes the pairing secrets, so the code can't be reused;
• inserts a join_request row for the receiver: { "deviceName": "…" }.

5. Joining a session

iPad                              Backend                         Receiver
 │ POST session-pair ───────────────►│ status=pairing               │
 │ subscribe row + signaling (1)     │── join_request ─────────────►│ shows Approve/Deny
 │ backfill select (2)               │◄──────── POST session-approve│
 │◄────── join_approved ─────────────│ status=connecting            │
 │ create pc, offer …                │                              │ /viewer/:id

Subscribe first, then backfill, so that join_approved can't be missed:

1. Subscribe to Realtime postgres_changes:
   • UPDATE on public.casting_sessions with filter id=eq.<sessionId>. Watch status.
   • INSERT on public.signaling_messages with filter session_id=eq.<sessionId>. Keep rows where recipient_role == "sender".
2. Once the channel reports SUBSCRIBED, backfill:

   GET https://sxhxcjoqxrauycxizing.supabase.co/rest/v1/signaling_messages
       ?session_id=eq.<id>&recipient_role=eq.sender&order=created_at.asc
   

3. Merge both streams and de-duplicate by row id.

RLS lets the sender read its session row and any rows addressed to sender whenever auth.uid() = sender_user_id. That includes after the session ends, so you can always see disconnected/expired and the final disconnect message.

join_approved payload:

{
  "senderAuthExpiresAt": "2026-10-20T12:40:00.000Z",
  "streamPrefs": { "autoQuality": true, "resolution": "1080p", "frameRate": 60 }
}

streamPrefs are the receiver's preferences only:

• resolution is "720p", "1080p" or "native";
• frameRate is 30 or 60;
• the iPad decides the final capture settings (the extension should cap at 1080p30, see §12).

If the receiver denies, you receive disconnect {"reason":"denied"} and the status becomes expired.

For reference, the receiver approves with POST session-approve {"sessionId", "approve": true|false, "streamPrefs"?} → {"sessionId","status":"connecting","senderAuthExpiresAt"} or {"sessionId","status":"expired"}.

6. WebRTC signaling

Roles:

• Sender (iPad) = offerer.
• Receiver = answerer, and it never sends media.

Row shape, as you insert it via REST POST https://sxhxcjoqxrauycxizing.supabase.co/rest/v1/signaling_messages:

{ "session_id": "<id>", "sender_role": "sender", "recipient_role": "receiver",
  "message_type": "offer", "payload": { … } }

RLS only accepts the insert if:

• you are this session's sender;
• your authorization hasn't expired (sender_auth_expires_at > now());
• sender_role is "sender" and recipient_role is "receiver";
• the session status is pairing, connecting or connected.

Payloads:

Rules:

• negotiationId: generate a fresh UUID for every offer. The receiver tears down its peer connection and builds a new one for each new negotiationId (this is how reconnects work). Candidates and answers with any other negotiationId must be ignored.
• Candidate buffering: candidates that arrive before setRemoteDescription(answer) are buffered per negotiationId and applied right after it.
• endOfCandidates: send it once ICE gathering completes (a nil candidate in peerConnection(_:didGenerate:) terms, or iceGatheringState == .complete). On receipt, call addIceCandidate with no candidate, or skip it if the SDK lacks that.
• De-duplication: by row id, across Realtime and backfill.

Limits (enforced by a database trigger):

• A payload may be at most 64 KB. Larger inserts fail with SQLSTATE 54000.
• A session accepts at most 200 signaling rows per minute, counting both sides. Beyond that, inserts fail with SQLSTATE P0429.
• Don't send one row per byte of trickle; one row per candidate is fine.

7. SDP offer/answer

• Configuration: bundlePolicy = .maxBundle and iceServers from ice-config.
• Sender: add one video transceiver (.sendOnly), plus an optional .sendOnly audio one (app audio). Put all tracks in one media stream id, because the receiver builds its view from streams[0].
• Receiver: answers with recvonly video and audio. It never sends media.
• Codec preference: order video codecs H.264 first, then VP8 as a fallback (RTCRtpTransceiver.setCodecPreferences). The reference sender logs the resulting m=video order. (Open-source Chromium builds have no H.264, so the logged order there starts with VP8; Google Chrome and iOS WebRTC include H.264.) On iOS, use RTCDefaultVideoEncoderFactory / VideoToolbox hardware encoding:
  • prefer Constrained Baseline (profile-level-id=42e01f) for the widest Chrome decode compatibility and lowest latency;
  • High (640c1f / 640c34) is supported by desktop Chrome and gives better quality per bit, but verify it on the target machines;
  • always keep packetization-mode=1.
• Encoder limits after setLocalDescription, via sender.parameters:
  • maxBitrateBps ≈ 4,000,000 for 1080p (2.5 Mbps for 720p);
  • maxFramerate 30;
  • degradationPreference = .maintainResolution for screen content.
• The receiver answers once per offer. A renegotiation is a new offer with a new negotiationId.

8. ICE

GET /api/public/ice-config (any signed-in user)

{ "iceServers": [ { "urls": ["stun:stun.l.google.com:19302", "stun:stun1.l.google.com:19302"] } ],
  "turnConfigured": false }

When TURN secrets are set on the server, a second entry { "urls": [...], "username": "…", "credential": "…" } is added and turnConfigured is true. TURN is not configured today, so connections across different networks or strict NATs may fail. This is not validated.

Trickle ICE: send each local candidate as an ice_candidate row as soon as it is generated, then endOfCandidates. The reference sender fetches ice-config once per broadcast and reuses it across reconnects; fetching it again before each new peer connection is also fine.

9. Session termination

POST /api/public/session-update

{ "sessionId": "<id>", "action": "connected" | "heartbeat" | "end" }

Other errors: 400 invalid_request; 404 not_found (not a participant, or the sender's authorization expired).

disconnect reasons that actually exist:

What each side observes:

• The iPad sees the status change to disconnected/expired and/or the disconnect row. It must stop capture and call finishBroadcastWithError with a user-readable message.
• The receiver shows "Your iPad has disconnected." for a disconnect message, or "The broadcasting session has ended." for a status change.
• Cron: sessions ended by the cleanup job get a status change but no disconnect row.

Best effort: on app or extension shutdown, call end (do not rely on it).

10. Sender authorization expiry

• Approval sets sender_auth_expires_at = now + 2h.
• heartbeat every 30 s while connected sets it to min(now + 2h, connected_at + 12h). If connected_at is not set yet, the cap counts from created_at. A session therefore can't last more than about 12 hours.
• Once expired:
  • RLS immediately rejects your signaling inserts;
  • session-update returns 404 not_found;
  • the cleanup job (every 5 minutes) moves the session to disconnected;
  • you can still read the row and messages addressed to you.
• After expiry, pair again with a new code.

11. Network interruptions

The sender drives recovery, matching src/lib/sender.ts:

• When iceConnectionState == .failed, or .disconnected for more than 5 s: close the peer connection, create a new one, use a new negotiationId and send a new offer.
• Backoff between attempts is 1 s, 3 s, 7 s, for at most 3 attempts. After that, give up and show a failure.
• A successful connection resets the counter.

The receiver's rules (src/lib/receiver.ts):

• It waits 8 s for a disconnected connection to recover on its own.
• After that, or when the connection fails, it shows "Reconnecting…" and waits 30 s for a fresh offer.
• It does this at most 3 cycles before showing Failed with a Retry button.
• A new offer always replaces the old peer connection.
• After a receiver page refresh on a connected session, it ignores old offers and waits for your next one. Re-offer when your connection drops.

App backgrounding: the Broadcast Upload Extension keeps running while the main app is in the background; capture continues. The main app may be suspended, so keep all WebRTC, signaling and heartbeats in the extension. When the user stops the broadcast from Control Center, broadcastFinished runs: call end there.

12. ReplayKit Broadcast Upload Extension guide

Lifecycle (RPBroadcastSampleHandler subclass):

• broadcastStarted(withSetupInfo:):
  • read the session id and tokens from the App Group / Keychain;
  • sign in with the shared session (setSession);
  • subscribe and backfill (§5), then ice-config;
  • create the peer connection and send the offer.
• broadcastPaused() / broadcastResumed(): frames stop and resume. Keep the peer connection; optionally send a black frame or nothing.
• broadcastFinished(): send disconnect, call session-update end, close the peer connection.
• processSampleBuffer(_:with:):
  • .video: screen frames → WebRTC;
  • .audioApp: app audio (optional audio track);
  • .audioMic: microphone; ignore it unless the user opted in.

Memory (~50 MB hard limit):

• Run WebRTC inside the extension with a lean configuration: one peer connection, no local renderer, no simulcast.
• Use the hardware H.264 encoder (VideoToolbox).
• Never copy pixel buffers: wrap the CVPixelBuffer from the CMSampleBuffer directly.
• Cap at 1080p30. Drop frames rather than queue them.
• Watch os_proc_available_memory().

Feeding frames:

• CMSampleBufferGetImageBuffer(sample) → RTCCVPixelBuffer(pixelBuffer:) → RTCVideoFrame(buffer:rotation:timeStampNs:) → videoSource.capturer(_:didCapture:).
• Set the rotation from the RPVideoSampleOrientationKey attachment (a CGImagePropertyOrientation):
  • .up → ._0;
  • .left → ._90;
  • .down → ._180;
  • .right → ._270.
• The receiver handles portrait and landscape automatically from the decoded dimensions.
• Use videoSource.adaptOutputFormat(toWidth:height:fps:) to cap the size.

App ↔ extension hand-off:

• Share the sessionId through an App Group (UserDefaults(suiteName:)).
• Store the access and refresh tokens in a shared Keychain access group (recommended), not in UserDefaults.
• Both processes may refresh the token, so the extension should write refreshed tokens back.

Starting:

• Show RPSystemBroadcastPickerView with preferredExtension = "<extension bundle id>" and showsMicrophoneButton = false.
• The extension cannot show UI. Report user-visible errors with finishBroadcastWithError(NSError(domain:…, userInfo:[NSLocalizedDescriptionKey: "The receiver ended the session."])).

13. Native implementation checklist

• Anonymous sign-in, persisted session, shared tokens via Keychain access group
• QR scanner + PIN entry → session-pair; handle 400 / 429 (Retry-After)
• Subscribe (row + signaling) before backfill; de-dupe by id
• Wait for join_approved; read streamPrefs; handle denied / expired
• Picker → extension; extension reads the session from the App Group
• Send-only H.264-first peer connection, max-bundle, encoder caps, maintainResolution
• Offer with a new negotiationId; trickle candidates + endOfCandidates
• Apply the matching answer; buffer early candidates; ignore other negotiationIds
• Heartbeat every 30 s
• Reconnect: failed or disconnected > 5 s → re-offer, 1 s/3 s/7 s ×3
• React to disconnected/expired status and disconnect rows → finishBroadcastWithError
• broadcastFinished → disconnect + end
• Rotation from RPVideoSampleOrientationKey; 1080p30 cap; stay under 50 MB

Swift skeleton — pseudocode / not implemented

// PSEUDOCODE — NOT IMPLEMENTED. Illustrates the contract only.
final class SampleHandler: RPBroadcastSampleHandler {
  var pc: RTCPeerConnection!
  var source: RTCVideoSource!
  var negotiationId = UUID().uuidString
  var seen = Set<String>(); var pending: [String: [Payload]] = [:]
  var attempts = 0

  override func broadcastStarted(withSetupInfo _: [String: NSObject]?) {
    Task {
      let shared = SharedStore.load()                    // App Group + Keychain
      try await supabase.auth.setSession(accessToken: shared.access, refreshToken: shared.refresh)
      await subscribe(sessionId: shared.sessionId)       // row UPDATE + signaling INSERT
      await backfill(sessionId: shared.sessionId)        // recipient_role=eq.sender
      startHeartbeat()                                   // every 30 s
      try await negotiate()
    }
  }

  func negotiate() async throws {
    negotiationId = UUID().uuidString
    let ice = try await api.get("ice-config")
    pc = factory.peerConnection(with: config(ice, bundle: .maxBundle), delegate: self)
    let tx = pc.addTransceiver(with: factory.videoTrack(with: source), init: .sendOnly)
    tx.setCodecPreferences(h264First(factory.supportedCodecs))
    let offer = try await pc.offer(for: constraints)
    try await pc.setLocalDescription(offer)
    applyEncoding(maxBitrate: 4_000_000, maxFramerate: 30, .maintainResolution)
    try await insert("offer", ["sdp": offer.sdp, "type": "offer", "negotiationId": negotiationId])
  }

  override func processSampleBuffer(_ sb: CMSampleBuffer, with type: RPSampleBufferType) {
    guard type == .video, let px = CMSampleBufferGetImageBuffer(sb) else { return }
    let rot = rotation(from: sb.attachment(RPVideoSampleOrientationKey))
    let frame = RTCVideoFrame(buffer: RTCCVPixelBuffer(pixelBuffer: px), rotation: rot,
                              timeStampNs: Int64(CMSampleBufferGetPresentationTimeStamp(sb).seconds * 1e9))
    source.capturer(capturer, didCapture: frame)          // no copies
  }

  func onRow(_ row: Row) {                                // from Realtime or backfill
    guard seen.insert(row.id).inserted else { return }
    switch row.type {
    case "answer" where row.nid == negotiationId: setRemote(row); flush(pending[negotiationId])
    case "ice_candidate" where row.nid == negotiationId: remoteSet ? add(row) : buffer(row)
    case "disconnect": finishBroadcastWithError(err("The receiver ended the session."))
    default: break
    }
  }

  func iceChanged(_ s: RTCIceConnectionState) {           // .failed or .disconnected > 5 s
    guard attempts < 3 else { return finishBroadcastWithError(err("Connection lost.")) }
    let delay = [1, 3, 7][attempts]; attempts += 1
    after(delay) { try await self.negotiate() }
  }

  override func broadcastFinished() {
    try? await insert("disconnect", ["reason": "sender_ended"])
    try? await api.post("session-update", ["sessionId": id, "action": "end"])
    pc.close()
  }
}

14. Status

• Native iPad broadcaster: NOT IMPLEMENTED.
• Web receiver validated with the browser test sender only (same machine, host candidates).
• Cross-network / TURN: not validated.