Reactive Lights and where to find them Part 2 of 2

2026-04-20

The Software Side of Things

In Part 1, I covered the hardware: soldering the LED strip, wiring it to the Xiao Seeed Studio ESP32-C6, and assembling everything on a breadboard. This post picks up right where it left off.

The rough plan was simple enough:

Capture a screenshot
Analyse the image and extract colors
Send those colors to the ESP32 to light up the LEDs
Do this as fast as possible

Simple on paper. Much more interesting in practice.

Choosing a Communication Protocol

The first real decision was: how does the laptop talk to the ESP32?

I considered a few options:

Serial over USB - simple, but ties the ESP32 to the laptop with a cable. Defeats the purpose of a wireless microcontroller.
HTTP - too much overhead. A full TCP handshake and HTTP headers for 156 bytes of color data, 10 times a second, is overkill.
WebSockets - better, but requires running a server somewhere.
MQTT - a lightweight pub/sub protocol designed exactly for this kind of IoT use case.

MQTT won. Here's why it fits so well:

The laptop acts as a publisher. It captures frames, processes colors, and fires off a small binary packet to an MQTT broker (I'm running Mosquitto locally). The ESP32 acts as a subscriber - it connects to the broker and receives those packets the moment they arrive. Neither side needs to know the other exists. The broker handles delivery.

The entire color payload for all 52 LEDs is 156 bytes coming from 52 zones * 3 bytes (R, G, B). At 10 frames per second, that's about 1.5 KB/s of network traffic. My local network barely notices it.

The Publisher: Laptop Side

The laptop script does three things in a tight loop: capture, process, publish.

Capturing the Screen

I used mss for screen capture. It's dramatically faster than Pillow's ImageGrab because it reads directly from the display server without going through Python image objects. The raw buffer gets converted to a NumPy array in a zero-copy operation:


screenshot = sct.grab(region)

frame = np.frombuffer(screenshot.raw, dtype=np.uint8).reshape(
    screenshot.height, screenshot.width, 4
)

This eliminates any need for decoding and read or write operations.

Mapping the Screen to LEDs

The LED strip runs horizontally across the front of my desk, left to right. So the mapping is straightforward: divide the screen into 52 equal vertical zones (columns), and each zone maps to one LED. The rightmost zone: LED 0. The leftmost zone: LED 51.

zone_width = width // led_count
zone = frame[:, x_start:x_end, :]   # full height, zone-width columns

The Color Extraction Problem

Here's where things got genuinely interesting.

The naive approach just averaging all the pixels in a zone produces terrible results in practice. Two problems showed up immediately:

1. Black letterbox bars. A lot of movies have black bars at the top and bottom of the frame. Include those pixels in the average and every LED gets dragged toward black, even when there's a vivid sunset on screen.

2. Subtitles and blown-out whites. White subtitle text, a bright window in a scene, an overexposed sky; all of these drag the average toward white and wash out the actual scene color.

A simple brightness threshold (if all channels < 20, exclude the pixel) handles the black bars, but does nothing for the whites. What I really needed was a way to filter out pixels that are unsaturated; too grey or too white to be representative of the scene's actual color.

The fix was to stop thinking in RGB and start thinking in HSV (Hue, Saturation, Value):

Hue is the color itself (red, green, blue, etc.). Its value lies on a color wheel, divided in 3 sections roughly corresponding to red, green, and blue going
Value is brightness. A pixel with V < 8% is too dark to be meaningful, so we excluded it.
Saturation is colorfulness. A pixel with S < 15% is too grey or white to be representative, so we excluded it.

A pixel must pass both tests to count toward a zone's mean. A subtitle is bright (high V) but has near-zero saturation (S ≈ 0), it is too bright hence excluded. A black bar fails on V, it is too dark hence excluded. A vivid red or teal passes both and is included.

Typically, you would compute HSV values for a give RGB triplet as a follow.

Value = max(R, G, B)
delta = V - min(R, G, B)
Saturation = delta / V  if V > 0  else 0
if delta == 0:
    Hue = 0   # Black, white, and grey have no hue
elif Value == R:
    Hue = 60 * (( (G - B) / delta) % 360)
elif Value == G:
    Hue = 60 * (( (B - R) / delta) + 2)
elif Value == B:
    Hue = 60 * (( (R - G) / delta) + 4)

Computing full HSV is expensive. But since we only need S and V (not the hue angle), we can skip the trigonometry and compute them directly from the RGB max and min channels:

V = max(R, G, B)
delta = V - min(R, G, B)
S = delta / V  if V > 0 else 0

This runs as a vectorised NumPy operation across the entire frame in one pass, before the per-zone loop. The resulting boolean mask is then sliced per zone, so the expensive part only happens once per frame, not 52 times.

If a zone has no valid pixels at all, for example a solid black letterbox bar, the corresponding LED is simply set to off (0, 0, 0). No colour is better than a wrong colour.

Timing

The capture loop uses a time.perf_counter-based fixed timestep rather than time.sleep():

deadline += FRAME_INTERVAL
sleep_for = deadline - time.perf_counter()
if sleep_for > 0:
    time.sleep(sleep_for)

The difference: if a frame takes 80ms to process and the allowed time interval is 100ms, the next frame starts in 20ms, not 100ms. Accumulated drift gets self-corrected every frame. In practice this gets me a consistent 8-9 FPS on my machine, which is smooth enough.

The Subscriber: ESP32 Side

The ESP32 runs CircuitPython and uses adafruit_minimqtt to subscribe to the MQTT topic. When a 156-byte payload arrives, it updates all 52 LEDs and calls strip.show() once.

Memory Allocation

CircuitPython's garbage collector runs synchronously. If it triggers mid-animation, you get a visible stutter. The fix is to avoid allocating new objects inside the message callback entirely.

The buffer is allocated once at startup:

_payload_buf = bytearray(LED_COUNT * 3)

And on every message, we copy into it rather than creating a new object:

_payload_buf[:] = message

Similarly, strip.show() is called exactly once per message, so all 52 pixels update in a single DMA burst.

Loop Timing

adafruit_minimqtt's loop() method accepts a timeout parameter that controls how long it blocks waiting for socket data. The default is 1 second, which is too long when frames arrive every 100ms.

Setting socket_timeout=0.05 and loop(timeout=0.15) means each loop iteration takes at most 150ms, which keeps pace with the publisher without burning CPU:

client = MQTT.MQTT(
    ...
    socket_timeout=0.05,
    recv_timeout=0.1,
)

# in the main loop:
client.loop(timeout=0.15)

The Result

After all of that, here's what the finished system looks like end-to-end:

Screen capture (mss)
    → NumPy HSV filter (one pass, full frame)
    → 52-zone vertical average (masked mean per zone)
    → 156-byte binary payload
    → MQTT publish (QoS 0, ~1.5 KB/s)
    → ESP32 receives, copies to buffer, calls strip.show()
    → LEDs update

The end-to-end latency is around 50–100ms. At 8–9 FPS, that's imperceptible, the lights feel genuinely reactive.

When I quit the publisher, it sends a zeroed payload (all black) with QoS 1 before disconnecting, so the LEDs turn off cleanly rather than freezing on the last frame.

What I'd Do Differently

A few things I'd revisit if I were starting over:

Dominant color instead of mean color. Averaging the RGB values works, but a zone with one vivid red object against a dark background gets diluted to a muddy pink. Extracting the dominant color, for example via a tiny k-means cluster or a histogram peak, would give punchier results. I haven't implemented this yet because it's slower, but the HSV filtering gets most of the way there by excluding the dull background pixels.

Temporal smoothing. The LEDs can flicker slightly during fast cuts or strobing scenes. A simple exponential moving average over two or three frames would smooth this out without adding noticeable lag.

Code

The full source is on GitHub: adityadhegde/reactive-lights.

laptop_publisher.py — screen capture, HSV filtering, MQTT publishing
esp32_subscriber.py — CircuitPython subscriber for the ESP32

The README has setup instructions, wiring details, and a tuning guide for the HSV thresholds.