I have a silly side project . What if I could synchronize audio sinks (speakers) across multiple machines (including hypervisors and their guests) so I could have synchronized A/V across multiple devices in the same room? Kind of like the Unifi Enterprise Audio/Video Bridge.
NoteThe AI section comes after all this context with Unifi and HDBaseT! It's interesting — trust me .
Unifi Enterprise Audio / Video Bridge
I saw this neat product line that Unifi is trying to establish itself in. The Audio / Video bridge brings HDMI and two balanced audio channels in and out with any device. I could have one of these in the basement connected to an Apple TV and another across several rooms and get a live low-latency video stream to each site! A rather neat unit — if set up in the ideal environment.
I have two of these and have experimented with them. Their implementation will work well for their target audience: stadiums and large promotional corporate displays that can afford all the other complimentary products — like a networking switch that costs as much as a beater car.
Consumer switches (including the not-enterprise switch lines from Ubiquiti) do not implement Precision Time Protocol (PTP). These A/V bridges use PTP to determine all the digital processing delays that occur between source and all sinks. Without PTP, it initially spasms with RGB static for the first minute. I suspect they have a fallback that uses Network Time Protocol and it takes a while to engage.
WarningIf you're sensitive to flickering lights, don't play the 4 second video below.
Besides the initial spasming lights, there's another issue I ran into: multicast flooding. The way Unifi A/V bridges support one source, multiple sinks, without the source having to duplicate its traffic is to use multicast. While multicast packets might not be broadcast to every leaf node, they did reach every switch even if the switch had no multicast subscriptions. For switches that had a 1Gbit link to my aggregate switch, such as the one between my Apple TV and the internet, it meant that it couldn't even play 240p youtube videos when the switch was continually flooded with nearly 1Gbit uncompressed video data.
The answer here isn't to upgrade all the switches and the links between them. It's to turn on IGMP snooping which lets switches track which leaf nodes are actually subscribed to multicast traffic and to only forward traffic through switches that have subscriptions.
"Why are you using 1Gbit links everywhere?" I'm not. I'm actually using 10Gbit SFP+ Direct Attach Copper between all my switches in the basement, including to the internet. The issue is how most rooms with multiple networked devices have a 1Gbit switch in between and the switch being flooded made the Apple TV unusable.
Thankfully these links are cheap within the same rack. They are too short to run inside buildings though.
Unifi's Enterprise Audio / Video bridge can work on gigabit just fine. It just takes the whole gigabit. Meaning, it is designed for direct point to point from receiver or transceiver through a PTP capable switch. Unless you're willing to pay up $4000 per zone, it doesn't look like running a local switch in each zone with an aggregate switch in a utility closet is possible. Every sink needs a point to point link. That said, at least these points can travel over fiber, whereas the incumbent technology is stuck on twisted copper pairs.
However, I don't think the Unifi Enterprise Audio / Video Bridge is for me, especially in my home theater uses. It does motion smoothing and absolutely ruins anime with subtitles. Also it gets really hot. I doubt they have ASICs inside to do energy efficient video processing.
HDBaseT - HDMI over Category Cable
The incumbent technology HDBaseT is something I already use in my home theater. However, despite using the typical cat5e, cat6a, etc. cables that you're familiar with, it cannot electrically connect with network switches. Every connection is point to point with other HDBaseT hardware. This is easy to satisfy within a single zone where only visuals are sent over cat5e.
Adding more zones — where the audience perceives one or more speakers — is not an issue either. As long as all the speakers in the zone are in sync, the audience will have a good experience, even if other zones are not in sync.
When you add more speakers in the same zone and each speaker has its own digital processing inline, you may get auditory dissonance known as a comb filter. Ever been in a Walmart, costco, or whatever where they have a bunch of TVs on and the audio sounds choppy and reverby? That's the comb filter.
Since each audio sink is unlikely to have its own audio delay adjustment technology, this has to be done at the switch — and for a price of several (even 6-12) thousand dollars. Suddenly the Unifi enterprise A/V switch looks price-competitive.
If a TV had to get replaced in a large office and several TVs were displaying the same video and audio source in the same zone, then the new TV has to be manually calibrated to have its audio in sync with all the rest. And, if for unfortunate reasons, the newest TV is the slowest of all the audio sinks in the zone, then all other audio sinks have to be recalibrated against the newest TV. And again, each connection from the TV to the switch is point to point.
If I were to commit to HDBaseT across my home, I would have to set up point to point infrastructure in each room.
My own Audio / Video bridge
As described on It's not the compute, it's the memory bandwidth, I'm working on my own A/V toolbox of sorts. And recently I found that HDMI out and USB DACs don't have their speakers in sync. How can I solve this without pushing calibration data onto the application? And, can I use multiple instances of this toolbox to synchronize within the same audio zone on a shared network?
For an audio-experienced person, musical or otherwise, tuning by ear can get you pretty close between two audio sinks. If they're emitting different tones, you can intuitively tell if they're on the same beat, so to say. Less so when the sounds smear…
CommentFrustrated? I still can't align these samples by hand — even if I click the solve button and then shift it out and back. Microsecond precision is just not humanely possible. That is what we need when more than two channels are deployed.
What if there are five or more channels? Pair-wise calibration is possible within a margin of error, but when you have multiple errors stacked together, the errors will make the listening experience less than ideal. That, and not every channel has the same frequency response. For example, a sub woofer won't resonate on the higher tones.
What alternatives are there to tones or chords or even a drum kit to synchronize audio channels? Turns out... noise! Uncorrelatable noise!
Calibration with Maximum Length Sequences (MLS)
You won't find much of this on YouTube. Maybe I'll change that. Maximum Length Sequences are periodic sequences over their maximum length and can be physically made using linear-feedback shift registers, in other words: electronically cheap random noise. Not suitable at all for cryptography, by the way.
The magic of MLS is how the noise is mathematically guaranteed to be unique from all other sequences by using an already discovered primitive polynomial like x^8 + x^4 + x^3 + x^2 + 1. The point being, it is cheap to compute too and every MLS is unique, provided their seed values are unique (and not all 0's). An MLS gives a bit sequence [0,1,1,0,1,...] which we can map to a wave form as [-0.5, 0.5, 0.5, -0.5, 0.5, ...], where 0.5 is the gain magnitude.
InfoEven if the sequence itself is unique, there is still a risk of partial overlap. By coincidence "wire" and "wired" sound very similar in my examples, even though they're processed by SHA-256 before going into the MLS generator.
As white noise, an MLS covers the full frequency spectrum with equal amplitude. This solves the frequency response issues that tones have, since a room or speaker or even the microphone can capture tones differently, and it means that even if MLS is stacked on top of each other.. the result is more noise instead of confounding constructive or destructive interference.
Given a method to create unique white noise, how can this be used for calibration?
You could scrub sample by sample against the reference to find where all the peaks line up. In code that would be:
def convolve(a, b): out = [0] * (len(a) + len(b) - 1) for i, av in enumerate(a): for j, bv in enumerate(b): out[i + j] += av * bv return out def best_correlation(sum_signal, channel_samples): best_index = None best_score = None best_corr = None for index, sample in enumerate(channel_samples): # Correlation is convolution with the sample reversed. corr = convolve(sum_signal, sample[::-1]) score = max(abs(v) for v in corr) if best_score is None or score > best_score: best_index = index best_score = score best_corr = corr return best_index, best_score, best_corr sum_signal = [1, 2, 3, 2, 1] channel_samples = [ [1, 0, -1], [2, 3, 2], [-1, -2, -1], ] best_index, best_score, best_corr = best_correlation(sum_signal, channel_samples)
However, do you see the issue? Convolving the samples in the time domain — that is, sample by sample over time — is O(n2). For 3 seconds of audio samples at 48khz, you have 144,000 samples... The square of that being 20,736,000,000 comparisons. Even asking you to slide this 144,000 times is too much to ask!
CommentIf you do manage to align it, did you notice how there's no ramp where it grows from 49.9% to 70% and finally to 100%? This property allows us to get near microsecond precision with sound. There's no guessing exactly where the alignment should be.
Ever heard of Fourier Transforms? That's what makes spectrograms (a rolling Fourier transform) possible! What about the Convolution theorem? Instead of looping over everything in the time domain with a convolution, we can do multiplications in the frequency domain and then apply the Inverse Fourier transform to bring the results back into the time domain.
Typically we don't directly use a Fourier Transform — which is O(n2) — instead we use a Fast Fourier Transform which is O(n × log(n)). Which is approximately 743k operations, a far better load to bear than 20t operations.
This is, unfortunately, where the visual intuition breaks down. While we have tools to examine the frequency domain, like Nyquist plots shown below, it can be difficult to grasp what is happening.
In the interactive example above, you might see a relationship if you adjust the delay slider on either channel. Some points in this scrambled spaghetti seem to stay anchored while other things rotate.
If you look at an individual channel and then compare to its conjugate, you'll see it flips upside down. This reverses time inside the frequency domain, apparently. Scan back up to the code above and you'll see the comment Correlation is convolution with the sample reversed. Since the convolution theorem gives us mathematical equivalence in the frequency domain, we have to fit our correlation problem into the convolution's toolkit, which is to reverse time on one side through inverting each frequency's phase.
If you multiply the sum signal — the thing our calibration microphone is picking up — and the conjugate of the channel's original audio in the frequency domain, you get a result that when converted back into the time domain — by the inverse Fourier transform — reveals the correlation weight with a single linear scan! All without having to compare samples 20t times!
This one (above) shows another sweep combined with an MLS source as noise. As you can see, when there's a low frequency spread in the sweep, there's more ambiguity around where the center actually is. However… if you click the button on the MLS channel, you'll see a single peak. No ambiguity at all. What do you think about that?
We can compare all the channels at once this way. The output we have to scan is per channel but the results can be plotted all on the same time axis. We can see exactly how delayed every channel is with a single sum recording!
Try tweaking the offset % bars (on the right side of each channel) to 0%. Note how no matter what you try, the sweep cannot lock on at 0 μSeconds.
Even with five simultaneous offset streams of noise, we can robustly identify the time delays between all of them with a single capture using this technique.
How did I even learn about this?
I have friends that work on space satellite infrastructure and one of them often remarks on all the details her side projects include — one of them being Maximum Length Sequences. That was a month ago and I only learned these things this week.
The and
of it all was knowing that this concept — synchronizing multiple devices over in a noisy environment with potentially measurable time delays — had been solved already. The fuzzy concept of streams of unique digits was in my head, but I could not link it up to this problem until I spent days chasing faulty assumptions.
I thought that I could use chords or pair-wise chords to synchronize channels over time, not realizing that signal smear and harmonics would give me noisy results or never even converge on a solution.
Who do I know that knows a lot about chords and harmonies? My Father. He performs barbershop and sometimes does some A/V work to help everybody learn their part.
He pointed out there's a neat trick where you can overlap delayed static and register peaks in the frequency domain that are different depending on the delay.
CommentSee how the bands look fuzzy and not well defined? It's the same problem again: I won't be able to reliably converge on the true delay from comb filter data.
Now I have at least an idea what won't work and why.
And so I ask :
Comb filtering
White noise
Multiple channels
Can test pair wise
Goal: find delays between each pair of channels
Goal: find maximally delayed channel
Goal: align all channels to maximally delayed channel
Comb filtering can identity valleys in frequency domain
Explain algorithm to calculate delays
Claude Sonnet 5 responds (truncated)
Pairwise delay estimation (per pair, time domain)
For channels i and j, cross-correlate:
R_ij(τ) = Σ_t x_i(t) · x_j(t+τ)
delay_ij = argmax_τ R_ij(τ)
With white noise excitation, use GCC-PHAT instead of raw cross-correlation — it whitens the spectrum before correlating, which sharpens the peak and kills reverberant smearing:
GCC-PHAT(τ) = IFFT[ X_i(f)·conj(X_j(f)) / |X_i(f)·conj(X_j(f))| ]
delay_ij = argmax_τ GCC-PHAT(τ)
Sub-sample accuracy: parabolic interpolation around the peak, or upsample before correlating.
And a bunch of other stuff I didn't ask for
At the time, I did not understand what in the world it presented to me. I struggled with math during university, especially calculus. Though, I've learned again and again that transforming problems from one domain into another is a reliable trick in computer science and math for optimal computation. So I dug deeper…
YouTube videos, more chats with , reading pages on Wikipedia
, more trial and error of putting in this vocabulary into Codex
to see what it does in practice with real speakers and microphones.
was my search engine to knowledge that I did not have. And,
could elaborate on finer and finer details that I could recombine into the bigger applicable concepts.
did not teach me. Experimentation and practice in real life taught me. Preparing the interactive panels (with
) in this article taught me. Writing about all this taught me.
Only after I could independently describe what I am doing and how it works have I gone back to my space satellite infrastructure friend for validation. Got a back.
When I tried to find relevant material to send to folks about what I was doing, it was either a 300 page audio system manual from the 90s or a highly scientific paper or something locked behind a scientific paywall, even though the document was written before I was born. That's part of why I am writing this article. Maybe it'll help someone else understand what maximal length sequences are and how they can be used in Generalized Cross-Correlation with Phase Transform (GCC-PHAT) to identify precise time delays between speakers, or even radio communications.