Allow for included streams in FragmentedSampleReader

[glennguy]

Hi @peak3d

There seems to be an assumption in the code that fragmented mp4 will always have separate video/audio playlists/segments (as it is for DASH and ISM) but I've found some examples of HLS that have video and audio in the 1 playlist (contained in the same .m4s segments) - there is no separate audio playlist available.

You can create a playlist like this with the following ffmpeg command: ffmpeg -i input.mp4 -hls_segment_type fmp4 -hls_flags single_file -hls_segment_filename main.mp4 -hls_list_size 0 out.m3u8

I've spent a long time in the last couple of weeks figuring out how it all works, doing lots of trial and error. My first thought was to parse the 'includedstream' as it's own separate stream but that would result in downloading the same segments multiple times. I think now that a solution lies in implementing the AddStreamType method in FragmentedSampleReader - my problem is that I'm at the limit of my abilities. I can see that firstly we will need to enable the additional track(s) i.e

AP4_Track *audioTrack = movie->GetTrack(TIDC[2]);
EnableTrack(audioTrack->GetId());

but before I spend anymore time I'm just looking to see if you could either point me in the right direction or if you have time/feel inclined to solve it yourself (in which case I hope any of this info helps).

Thanks for reading

Glenn

[peak3d]

You are right, for mp4 container internal audio stream is not implemented. Your code snipped seems to be fine, b.t.w, I'm not quite sure if there are other conditions checking for internal / container_hls in the code, but should be solveable with your above code.

[glennguy]

I'm hoping for a bit of guidance here. I've tried so many things and this is as far as I've got: https://github.com/glennguy/inputstream.adaptive/commit/47643a7bb8764b4bb72b1bfa98c589911e1e88a9

We now:

• allow additional tracks to be enabled in the mp4 via AddStreamType
• record which stream types are enabled and their streamids in m_typeMap (same as TSReader does)
• use the ReadNextSample override here in ReadSample() which will read the next sample in the data of any enabled track, and update m_trackId to the track which the sample came from

I've verified that the above is working nicely by injecting additional logging statements - additional track does get enabled in AP4LinearReader, additional streamid created in Kodi and DemuxRead receiving the samples from the correct streams.

I have updated any reference to m_track to now get the track belonging to m_trackId. I'm unsure if this is causing any issues but I've also tried various combinations of m_trackId and m_track->GetId() throughout FragmentSampleReader without any luck. Either the stream fails after the first chunk or it skips every second chunk, no audio in any case.

Another though is does the CodecHandler have anything to do with it? Do both streams need to have a separate CodecHandler (or way to return information from GetInformation()?

Here's a debug log with ffmpeg log enabled: kodi.txt

Thanks heaps Glenn

[peak3d]

from gut feeling I would say, one Reader, but multiple CodecHandlers (one per track).

[glennguy]

Ok excellent, thankyou! I'll figure out how to allow for that

[gizmocuz]

There is also a fragmented HLS example here:

https://bitmovin.com/demos/hls-fmp4

I don't think this is supported yet ? I suspect the tag 'EXT-X-MAP' needs to be implemented as well

[matthuisman]

another test stream if needed: https://github.com/xbmc/inputstream.adaptive/files/6071672/no_audio.strm.txt

[mfu-mcosys]

Same problem here. Peertube use "-hls_segment_type fmp4" for ffmpeg, i changed it to mpegts but this breaks other stuff (i think because of #EXT-X-VERSION:6 vs #EXT-X-VERSION:7)

So.. any updates?