bilaldemir.dev

Dropping Frames on Purpose: Low-Latency Streaming Without Queues

DisplayBridge turns another device into a second monitor for a Mac. An Android tablet over USB or Wi-Fi, an iPad, another Mac, or a Linux or Windows machine through mpv. It creates a virtual display on the Mac, captures it, encodes it with hardware H.265, and streams it to the other device. Touch and pen input travel back the other way. The goal is under 16 ms from the Mac's screen to the tablet's, which is one frame at 60 Hz.

VirtualDisplay -> ScreenCapturer (IOSurface) -> VideoToolboxEncoder (H.265)
  -> PacketFramer (28-byte header + NAL) -> TCP or USB (AOA)

The project is still young: the iOS sink hasn't been built with Xcode yet, and the Windows sink has only been tested under Wine. But the core design questions are settled, and the most important one is about queues.

Why a queue is the wrong tool

The instinctive design for a capture-encode-send pipeline is a queue between each stage. Capture pushes frames, the encoder pulls them, the sender pulls encoded frames. Queues smooth out bursts, and nothing gets lost.

For a live display, "nothing gets lost" is the problem. If the network stalls for 100 ms, a queue fills up with six frames that are all already out of date. When the network recovers, the sender works through them in order, and the tablet shows the past for another 100 ms. Every hiccup adds latency that only goes away once the queue drains. For a monitor you're dragging windows on, an old frame is worse than no frame.

So DisplayBridge follows one rule: always send the newest frame, drop stale ones, and never queue.

Two tiny primitives

The rule is implemented with two types, each a few lines behind a mutex. The first is a gate that allows at most one frame in flight on the transport:

pub fn try_enter(&self) -> bool {
    let mut g = self.inner.lock().unwrap();
    g.total_frames += 1;
    if g.busy {
        g.dropped_frames += 1;
        return false;
    }
    g.busy = true;
    true
}

pub fn leave(&self) {
    let mut g = self.inner.lock().unwrap();
    g.busy = false;
}

If the gate is busy, the caller drops the frame. It doesn't wait, and it doesn't buffer. The counters exist so that dropped frames show up in metrics instead of disappearing silently.

The second is a single-slot holder for the most recent encoded frame. The encoder overwrites it, and the sender takes whatever is there when it's ready:

/// Stores a new pending frame, overwriting any previous (unsent) one.
pub fn set(&self, packet: Vec<u8>, capture_time_ns: u64) {
    let mut slot = self.inner.lock().unwrap();
    *slot = Some(Slot { packet, capture_time_ns });
}

/// Takes the pending frame, clearing the slot.
pub fn take(&self) -> Option<(Vec<u8>, u64)> {
    let mut slot = self.inner.lock().unwrap();
    slot.take().map(|s| (s.packet, s.capture_time_ns))
}

Together they let the encoder work ahead of the network without building a backlog. When the link is fast, every frame goes through. When it's slow, the sender skips straight to the newest frame each time it gets a chance. Latency stays at roughly one frame plus the network, whatever happens to throughput.

Because the policy is so small, it's easy to test exactly:

#[test]
fn gate_allows_one_and_drops_while_busy() {
    let gate = PipelineGate::new();

    assert!(gate.try_enter());   // first frame acquires the gate
    assert!(!gate.try_enter());  // while busy, further frames are dropped
    assert!(!gate.try_enter());
    assert_eq!(gate.stats(), (3, 2));

    gate.leave();
    assert!(gate.try_enter());   // after leaving, the next frame goes through
}

One core, many platforms

The Mac side is Swift, the Android sink is Kotlin, and the desktop sinks are Rust. Each one needs the same protocol, handshake, heartbeat and pairing logic. Writing that four times is how platforms drift apart, so the shared logic lives in a Rust workspace that the README calls the single source of truth:

Much of this started life in Swift and Kotlin, and the Rust modules say so: the packet framer was ported "byte-for-byte" from the originals. The Android sink still has its own framer for now.

A pure state machine. The session core has no I/O, no clock and no threads. The native shell feeds it events (bytes arrived, a tick, a disconnect) and gets back commands (send this, start capture, close). Even timestamps are passed in with the event. A heartbeat that pings every 3 seconds and gives up after 10 seconds of silence can then be tested without sleeping, by feeding in fake times.

A C ABI, not a binding generator. Tools like UniFFI generate nicer Swift and Kotlin APIs, but a plain C ABI is the one thing every target platform can consume. Every entry point checks its pointers for null and is wrapped in catch_unwind, so a Rust panic never unwinds into Swift or Kotlin. That would be undefined behaviour. The C header is generated by cbindgen on every build.

A wire format you can read

Every packet has a 28-byte header followed by its payload: a 4-byte magic value (DBRG), the packet type, a sequence number, a capture timestamp and the payload length. The decoder rejects anything without the magic value, and caps the payload at 16 MB. That way a corrupted length field produces an error instead of a 4 GB allocation.

A pairing code, with an honest threat model

Over the network, a device has to enter the code shown on the Mac before it gets a picture or can send input. The comparison is constant-time, and failures are counted in a sliding one-minute window shared by all connections:

/// Caps guessing a 6-digit code at 5 tries a minute.
pub const MAX_FAILURES: usize = 5;
pub const LOCKOUT_NS: u64 = 60_000_000_000;

pub fn codes_match(expected: &str, got: &str) -> bool {
    let (a, b) = (expected.as_bytes(), got.as_bytes());
    a.len() == b.len() && a.iter().zip(b).fold(0u8, |acc, (x, y)| acc | (x ^ y)) == 0
}

The README is explicit about what this doesn't do. The stream isn't encrypted, and the pairing code is sent in the clear. It keeps other devices on your network out. It doesn't protect against someone who can already read your traffic. On a network you don't trust, use USB. Stating that plainly is better than implying a security level the project doesn't have.

A few other details

Takeaway

In real-time pipelines, decide what happens under backpressure before you write the pipeline. For a live display, the answer is to drop frames, and the code for that is smaller than any queue.

← All posts