video-dev
Improve Live Subtitle Synchronization with Intelligent Tolerance for HLS Streams
Hello. I encountered an issue where the VTT file doesn’t render at all during live subtitle playback. I resolved the issue using the following logic, but I believe it would be beneficial if hls.js included a similar mechanism. Therefore, I’d like to suggest this functionality as a feature request.
Is your feature request related to a problem? Please describe.
Yes, there's a critical issue with live HLS subtitle rendering that causes complete subtitle failure in real-time streaming mode.
Problem Details:
- Live HLS streams with timeMachine=false (non-DVR mode) experience intermittent subtitle failures
- Subtitles completely disappear during playback, affecting user experience significantly
- Issue occurs when subtitle fragments are slightly ahead of media playback position (typically 40-500ms)
Technical Root Cause: The current SubtitleStreamController.onSubtitleTrackLoaded() uses zero tolerance when matching subtitle fragments:
const foundFrag = findFragmentByPTS(null, fragments, mediaCurrentTime, 0); // tolerance=0
if (!foundFrag) {
this.warn('Subtitle playlist not aligned with playback');
track.details = undefined; // Immediate subtitle track disable
}
Timing Analysis:
- DVR Mode (Working): Long historical buffer (~2942s), delta: +0.605 (media leads subtitle) ✅
- Live Mode (Broken): Short sliding window (~33s), delta: -0.352 (subtitle leads media) ❌
Example failure case: media=35.904s, firstFragStart=36.256s → 352ms gap causes zero-tolerance failure
Describe the solution you'd like
- Apply Intelligent Tolerance: Replace hardcoded zero tolerance with configurable maxFragLookUpTolerance (default 0.25s):
const lookupTolerance = Math.max(this.config.maxFragLookUpTolerance ?? 0, 0);
const foundFrag = findFragmentByPTS(null, fragments, mediaCurrentTime, lookupTolerance);
- Multi-Stage Validation System: Instead of immediate failure, implement graceful handling:
if (foundFrag) return; // Success
// Wait if subtitle leads playback
if (mediaCurrentTime + lookupTolerance < fragWindowStart) {
this.log('Subtitle leading playback; waiting...');
return;
}
// Defer if within subtitle window
if (mediaCurrentTime - lookupTolerance <= fragWindowEnd) {
this.log('Subtitle lookup deferred...');
return;
}
// Only warn for genuine misalignment with detailed info
// ⭐ Alternatively, not disabling the track might also be a good option. Setting track.details = undefined is too risky.
this.warn(`Subtitle playback mismatch: media=${mediaCurrentTime.toFixed(3)}
tolerance=${lookupTolerance.toFixed(3)}`);
- Proactive Fragment Loading: Enhance doTick() to prefetch first subtitle fragment when timing gaps exist:
if (!foundFrag && fragments.length &&
targetBufferTime + lookupTolerance < fragments[0].start) {
foundFrag = fragments[0]; // Prefetch first fragment
}
The issue occurred because the VTT file lacks the X-TIMESTAMP-MAP tag, but I believe we should still implement a fallback to completely disable the track. What do you think?
Additional context
AS-IS
sequenceDiagram
participant Video as Video currentTime
participant Subtitles as Subtitle fragments
participant Controller as SubtitleStreamController
Video->>Controller: tick()
Controller->>Subtitles: findFragmentByPTS()
Controller->>Controller: warn "subtitle playlist not aligned"
Controller->>Subtitles: track.details = undefined
Controller-->Video: wait for next playlist refresh
TO-BE
sequenceDiagram
participant Video as Video currentTime
participant Subtitles as Subtitle fragments
participant Controller as SubtitleStreamController
Video->>Controller: tick()
Controller->>Subtitles: findFragmentByPTS()
alt fragment found
Controller->>Video: continue normal load
else fragment leading (media < firstFrag - tolerance)
Controller->>Controller: log waiting
Controller-->Video: wait for next tick
else fragment within tolerance
Controller->>Controller: log deferred
Controller-->Video: retry next tick
else fragment behind window
Controller->>Controller: warn mismatch
Controller->>Subtitles: drop track.details
end
Issue Logs
[log] > [subtitle-track-controller]: Loading subtitle 0 "한국어" lang:kor group:subs age 4.2 https://test.akamaized.net/qa/hls_chunklist.m3u8
[log] > [subtitle-track-controller]: Subtitle track 0 "한국어" lang:kor group:subs loaded [3505-3512]
[log] > [subtitle-track-controller]: live playlist 0 REFRESHED 3512--1
[log] > [subtitle-track-controller]: reload live playlist 한국어 in 3854 ms
[log] > [subtitle-stream-controller]: Subtitle track 0 loaded [3505,3512][part-3512--1],duration:32.083334000000036
[log] > [subtitle-stream-controller]: Setting startPosition to -1 to start at live edge 181.33688745117078
[log] > [subtitle-stream-controller]: @@ Subtitle offset media=176.283 firstStart=179.843 delta=-3.560
[log] > [subtitle-stream-controller]: @@ foundFrag null
[warn] > [subtitle-stream-controller]: Subtitle playlist not aligned with playback
Uncaught TypeError: Cannot read properties of undefined (reading 'type')
at Rr (hls-player.min.mjs:1:75612)
at t.getState (hls-player.min.mjs:1:73434)
at n.doTick (hls-player.min.mjs:1:400060)
at r.tick (hls-player.min.mjs:1:88247)
at n.onSubtitleTrackLoaded (hls-player.min.mjs:1:398510)
at o.emit (hls-player.min.mjs:1:5391)
at o.emit (hls-player.min.mjs:1:536313)
at o.trigger (hls-player.min.mjs:1:536371)
at t.handlePlaylistLoaded (hls-player.min.mjs:1:457960)
at t.handleTrackOrLevelPlaylist (hls-player.min.mjs:1:455257)
const prevFrag = curSNIdx > 0 ? fragments[curSNIdx - 1] : undefined;
const prevFragState = prevFrag
? this.fragmentTracker.getState(prevFrag)
: null;
It seems that the following revisions are necessary.
const prevFrag = fragments[curSNIdx - 1];
const prevFragState = this.fragmentTracker.getState(prevFrag);
I think track.details = undefined should be removed and the warning downgraded to a log message like subtitle playlist ends @{details.fragmentEnd} with currentTime @{media.currentTime}.
Loading the first fragment foundFrag = fragments[0] when segments are ahead would also be acceptable. But, if they are ahead with no overlap of the playhead, that suggests an issue with their timing.
Fix is up in #7626. branch test: https://bugfix-low-latency-webvtt-pa.hls-js-4zn.pages.dev/demo/