Seek / Random access using PTS / ctts for fmp4

[daramousk]

Is your feature request related to a problem? Please describe.

Hi,

I have some long running streams (weeks) for which discontinuities exist (ffmpeg node is on site and it might go offline for a number or reasons). Using ffmpeg I generate a m3u8 with fmp4 fragments and I timestamp every a/v frame with the epoch timestamp as I need to know exactly when certain events took place.

While this works fine (ffprobe -show_packets returns pts_time=1752460713.324687) and I can seek ok, using ffmpeg seeking in the user agent becomes an issue for me.

I need to be able to seek using that pts_time, but I noticed the following:

1. I would expect fragment.startPTS/fragment.endPTS to have the actual timestamps, but they seem to be overwriten by fragment.start, fragment.end and always start from 0. I would expect the startPTS/endPTS to be taking their values from the ctts box.

I have gone on and integrated mp4box.js (by integrated I mean change hls.js to extract raw data (ArrayBuffer) not just for the initSegment but for all of them) and then inspect the first frame of the first elementary stream, get the pts time and then put it on every fragment.

Before I continue this path, am I thinking correctly here? Shouldn't this value be already included on fragment.startPTS and fragment.endPTS?

Describe the solution you'd like

frag.startPTS, frag.endPTS and everywhere PTS is mentioned to have the actual PTS from the elementary streams That would make a seekToPTS implementation straightforward

Additional context

My use case is that I have created a timeline react component with trick play that have time of day on its X axis, and people click to navigate to a specific time of day. This works fine when there are no discontinuities but when they are I cannot just do seekpts - start_pts because it will be off by the sum of the duration of the discontinuities.

[robwalch]

startPTS, endPTS, and the like have not and cannot be changed for span of hls.js version 1.x. There are integrations that depend on these timestamps being relative to the HTMLMediaElement timeline.

This has always been possible by using initPTS. Use initPTS to find the offset from these values to the media presentation timestamps.

You can copy and track these offsets in INIT_PTS_FOUND:

const domainOffsets = {};

hls.on((Hls.Events.INIT_PTS_FOUND, { frag, initPTS: baseTime, timescale }) => {
  // keep track of this somewhere
  const offset = baseTime / timescale;
  const discontinuityDomain = frag.cc;
  domainOffsets[discontinuityDomain] = offset;
});

Or, read them from the streamController once media has been parsed.

Here's an example of doing that and seeking to a PTS value:

const currentFrag = hls.streamController.currentFrag;
if (currentFrag) {
  const initPTS = hls.streamController.initPTS[currentFrag.cc];
  if (initPTS) {
    const { baseTime, timescale } = initPTS;

    // seek to "900000"
    hls.media.currentTime = (900000 - baseTime) / timescale;

    // this fragment's starting PTS
    currentFrag.start_pts = currentFrag.startPTS * timescale + baseTime;
  }
}

You could replace currentFrag with any fragment, but at least one fragment in the same discontinuity domain (with the same cc) would have had to have been parsed to get its pts.

[robwalch]

My use case is that I have created a timeline react component with trick play that have time of day on its X axis, and people click to navigate to a specific time of day.

In that case you should be using PDT (EXT-X-PROGRAM-DATE-TIME) in your HLS playlists and for application seek logic. Hls.js has hls.playingDate which returns the js Date (time) of the playhead. Seeking is then just a matter for getting the difference between that date and currentTime and seeking to the desired date offset. You can also get the epoch datetime milliseconds of any fragment with fragment.programDateTime to do the same but account for inconsistencies, drift or discontinuities in EXT-X-PROGRAM-DATE-TIME.

[daramousk]

In that case you should be using PDT (EXT-X-PROGRAM-DATE-TIME) in your HLS playlists and for application seek logic. Hls.js has hls.playingDate which returns the js Date (time) of the playhead. Seeking is then just a matter for getting the difference between that date and currentTime and seeking to the desired date offset. You can also get the epoch datetime milliseconds of any fragment with fragment.programDateTime to do the same but account for inconsistencies, drift or discontinuities in EXT-X-PROGRAM-DATE-TIME.

This is the first route I took and my implementation was the following:

1. Given an epoch timestamp (lets call it seekEpochTime), export and use the findFragmentByPDT to find the fragment that contains the sample
2. Assuming the EXT-X-PROGRAM-DATE-TIME is equal to the pts of the first sample of the segment (the RFC seems to be implying that this is the case? cf. https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis#section-4.4.4.6) I can then easily do something like video.currentTime = currentFragment.startPTS + (seekEpochTime - currentFragment)

This is always off by a few seconds (always less than the duration of the fragment). This will not work for us as we need accurate (subsecond) seeking. I feel like my assumption that the EXT-X-PROGRAM-DATE-TIME is exactly equal to the pts time of the first sample of the segment is wrong; could that be the case? That seems to be supported by looking at the ffmpeg source code https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/hlsenc.c#L2973 (take this with a grain of salt as I have minimum knowledge of a/v programming)

As for the initPTS, that was indeed another route I had taken, however there I am facing an issue here:

1. Hls.Events.INIT_PTS_FOUND returns values that do not make sense to me;

I am not sure on how to interpret this baseTime value to get the pts_time as given by ffprobe; This is what ffprobe returns as a start value

With that being said, while I understand that the pts values on the fragments cannot be changed in a minor version, shouldn't their value be fetched from the elementary streams themselves verbatim? Basically from my (albeit limited) understanding, everywhere pts is mentioned, it should be fetched directly from the underlying a/v material.

[robwalch]

This is always off by a few seconds

That should not be the case unless there is an issue with your playlists program date times or your currentTime calculation.

PDT is mapped to the start of the segment it precedes.

playingDate can be used to get the date time at currentTime, and thus can also be used to seek relative to the current position:

hls.media.currentTime += (seekToDate - hls.playingDate) / 1000;

Hls.Events.INIT_PTS_FOUND returns values that do not make sense to me;

Can you file a bug with an example HLS sample (I can't remotely troubleshoot debugger screenshots)?

[daramousk]

Of course Rob,

Find a short HLS session here: https://drive.google.com/file/d/1QXs7wO2TsNPYNumB39pXZNT5jQ7OApjw/view?usp=sharing

and here is my fork of hls.js where I attempt to do seeking by tweaking the /demo application https://github.com/video-dev/hls.js/compare/master...daramousk:hls.js:pts

Given an input of 1753251200000 I would expect it to land on 4:13:20 yet it goes on 4:13:15