Syncing audio to a clock that ticks once a second

When you cast a video over DLNA, the sound is baked into the stream and comes out of whatever you cast to. Usually that is what you want. Sometimes you want the sound somewhere else: the picture on a screen across the room with the audio in your headphones so you do not wake the house, or a film on goggles strapped to your face without the room hearing it. Either way the audio has to come out of the PC and not the receiving device.

GogglePlayer, an app I built for streaming over DLNA, does that. It has an option to keep the video going to the renderer and play the audio on the PC instead, out to a headphone jack, a USB DAC, or Bluetooth earbuds etc. I built it around video goggles, but the renderer can be any DLNA receiver on the network. The split is one toggle. Making it not sound awful took most of the work.

The challenge is when you split the two then you end up with two clocks. The renderer runs video on its clock, the PC runs audio on its own, and nothing holds them together. Off by enough and the mouth moves a beat before the line lands, the whole film, the kind of wrong you cannot stop hearing.

You cannot just ask where the video is

The obvious approach is to ask the renderer where it is in the file and line the audio up to that. DLNA has a call for exactly that. The trouble is that a lot of renderers answer only to the nearest whole second, HH:MM:SS with no fraction, and audio sync goes wrong around 40 ms, so a one-second answer is useless. Some do not answer at all on a live stream and report zero the entire time. (The goggles I was testing on did both, which is how I learned to stop trusting the readout.)

The value is a dead end but the edges are not.

When the reported second flips from 7 to 8, the flip lands at a real instant, and at that instant the true position is exactly 8.000. The rounding throws away the fraction, but the moment the number changes still carries it. Poll fast enough to catch the flip and you have a precise pairing: this wall-clock instant is that exact media time. A handful of those pin down the only number you need, the wall-clock moment the file was at zero, and from there position is just now minus that anchor, worked out locally to the millisecond between polls.

It is the same move as clock recovery or a software PLL. The signal is coarse but its timing is clean, so you lock to the timing and drop the rest. A bit of smoothing rides out the poll jitter, but the edges are the whole idea.

A precise position is not enough

The two clocks do not run at the same speed. A PC sound card and a playback device's internal clock are both nominally 44.1 kHz the way two cheap watches are both sixty seconds to the minute: close but not equal. The error is constant and one-directional, so it stacks up. 0.5% sounds like nothing until you spread it across a two-hour film, where it adds up to 36 s of drift. Aligned at the titles, gone by the third act.

You could keep re-aligning: catch the drift, jump the audio back, repeat. It works and it sounds bad, because every jump is a seam you can hear, a click or a stutter every couple of minutes. A slow drift swapped for a regular flinch.

The version that does not flinch corrects speed instead of position. If the audio runs a touch fast, play it a touch slow. Resample it by a hair so it stretches just enough to cancel the skew. Done continuously and gently it is inaudible, because most of us do not hear a 0.3% pitch shift that we are not listening for, and the drift never builds over time, since you are matching the rate rather than catching the gap after it opens.

You do not know the skew up front. It depends on the exact hardware on both ends, so you find it while the file runs and keep finding it, because a network hiccup moves the target. That is a control problem.

The loop that hunts, and the loop that holds

The tool for a control problem is a feedback loop: measure the error between audio and video, feed it to a controller, let the controller set the playback rate. A positive error slows it down, a negative one speeds it up. A standard PI-controller, the kind that holds a thermostat at a certain level, aimed at the gap to drive it to zero.

The first one I built worked and was unusable. It found the right speed, overshot, corrected back too hard, overshot the other way, and oscillated like that indefinitely, clamp to clamp, on a ~100 s cycle you could hear: the audio sliding out, hauling back, sailing past, again and again. Shipping a correct control loop that makes the problem audibly worse is a particular kind of bad day.

The cause is that the thing being controlled is already an integrator. Rate sums into position, and an integral controller on top of an integrating plant rings. The fix is less force. You detune it until it is sluggish, so it eases onto the steady skew and parks there instead of sprinting at the target and blowing past. It locks slower, close to a minute on the first play, which looked like a regression right up until it locked once and then held dead flat for two hours.

Two details paid for themselves. The loop keeps what it learns across a seek, so scrubbing does not cost another minute of settling. And the learned skew is saved per device, so the next file on the same renderer and headphones starts already close and locks in seconds. Underdamped hunts, overdamped holds, and most of the work was getting from the first to the second.

Where it gives up

It tracks the steady gap between two clocks and holds it, and on a decent network you set the picture-and-sound offset once and forget the machinary is there. Small drift it trims out in the rate, where you cannot hear it. A jump too big to trim gently, the kind a network rebuffer causes, it does not chase slowly: past a threshold it stops trimming and snaps the audio back in one move on its own. That absorbs a one-off stall. What it cannot absorb is a connection that keeps stalling, where the snap just lands again and again, and no control loop beats a saturated link. There is a Resync to force a re-lock by hand, surfaced as a banner after a stall, but past a point the real fix is a cable.

That is the whole of it. A once-a-second readout turned into a smooth clock by timing its edges, and a deliberately lazy loop that matches two clocks' speeds instead of chasing the gap between them. GogglePlayer runs all of it in the background, whether you are casting to a screen across the room or a pair of goggles.