When tunneling is enabled, seeking to the end causes playback to be stuck, not reaching STATE_ENDED

[sruditsky]

We are using ExoPlayer 2.18.7 and Android API level 30. Our application sometimes requires calling ExoPlayer.seekTo with the positionMs equal to the duration of the content.

When we use non-tunneling mode this works as we expect it to, i.e. player gets into STATE_ENDED and the playback stops. However, when the same is done in the tunneling mode the playback gets stuck forever.

Analysis of what is causing this shows that under such conditions ExoPlayer starts feeding video decoder samples starting from the the last keyframe, but it does not feed any samples to the audio decoder. So the video decoder accepts several samples and then dequeueInputBuffer starts failing. It is failing because this is tunneling mode ate there are no audio samples to associate the video sample with.

The question is is such seeking to the content duration a valid operation? If "yes" then how should it work with tunneling mode? Or should ExoPlayer the call to seekTo fail?

A similar question is: let's assume we have a content which does not have any audio for the few last seconds (but does have video) and let's say the app is seeking to such a position with video but without audio. How should such a seek work in the tunneling mode (given that video cannot be synchronized without audio)?

[tonihei]

Tunneling is known to be a bit unreliable in some cases because it forwards a lot of the implementation details to OEM specific code Android cannot verify or control. This sounds like one of these cases. In theory, the end of the audio track should be sufficient to signal that all remaining video samples should be output. If you build locally, could you try enabling this workaround to see if it makes a difference? We had previous issues with end-of-stream signaling on some devices.

[sruditsky]

@tonihei My understanding of CodecNeedsEosPropagation is that some decoders do not propagate BUFFER_FLAG_END_OF_STREAM (EOS) from the input buffer to the output buffer and need ExoPlayer's help in doing so.

However, I am talking about a situation when the playback never reaches the state in which ExoPlayer submits a buffer with EOS neither to video nor to audio decoder.

When in tunneling mode seekTo(duration) is called ExoPlayer submits no input buffers (including no EOS buffer) to the audio decoder but does submit video buffers starting from the last keyframe. Video decoder gives ExoPlayer all the available input buffers, ExoPlayer fills them and submits back to the decoder and at this point the video decoder gets stuck and everything stops because the video decoder does not have any more input buffers and the ones which are submitted already are not being processed because in tunneling mode the buffers are waiting for the timing coming from the audio side.

When I repeat the same but with seekTo(duration - 20 msec) ExoPlayer submits the same video buffers to the video decoder as in the previous case but it also submits the last audio buffer and an EOS buffer to the audio decoder and in this case video decoder is not getting stuck and continues to process the video buffers up to EOS as expected.

Maybe the problem is that ExoPlayer needs to send EOS to the audio decoder even in the case when there is no more audio in the stream, i.e. when seekTo(duration) is called.

[tonihei]

Interesting, the CodecNeedsEosPropagation was just a quick attempt to see if your issue can be solved by a similar approach. Thanks for debugging it further to see what's actually happening.

When in tunneling mode seekTo(duration) is called ExoPlayer submits no input buffers (including no EOS buffer) to the audio decoder

I think the audio decoder is actually irrelevant for this setup and only what happens in DefaultAudioSink (or rather the platform AudioTrack) really matters for the A/V sync behavior. If you have a reproducible setup, could you check whether DefaultAudioSink.playToEndOfStream is called and whether isAudioTrackInitialized returns true in there? If yes, it will be a shortcoming of how tunneling is designed and there is probably not much we can do realistically. If no, we may be able to workaround it by ensuring the AudioTrack is initialized and stopped even if there is no actual audio data to play.

[sruditsky]

@tonihei Yes, I checked this. After the seek DefaultAudioSink.playToEndOfStream is called in a loop and isAudioTrackInitialized always returns false.

[tonihei]

Thanks for the confirmation. I think the best thing to try out is creating the audio track (and then stopping it immediately), as this may signal the end of playback to the video codec as well.

Could you add

if (tunneling && !isAudioTrackInitialized()) {
  try {
    initializeAudioTrack();
  } catch (InitializationException e) {
    throw new WriteException(/* errorCode= */ 0, e.format, e.isRecoverable);
  }
}

to the beginning of playToEndOfStream()? Not fully sure it helps, but at least it's worth a try.

Aside: Did you observe this problem on specific devices only or multiple devices?

[sruditsky]

I tried this. After this block isAudioTrackInitialized is returning true, but this did not change the problem behavior at all.

[tonihei]

Hmm, okay, thanks for trying this out in any case! I believe this may mean it's not possible to handle this case nicely in a simple way (=with showing the last frame) when tunneling is enabled.

I can imagine some more involved fixes that could help, for example:

• Sending the last few milliseconds of audio to play as well (although this may be not the ideal user experience either)
• Artificially playing silence to for a short duration.
• Not attempting to play the last frame at all in this case and forcing the player to move to the ENDED state immediately.

In your app, do you control the UI implementation? If so, do you think it's feasible to 'solve' this by never truly seeking to the end of the media but slightly before? This is similar to my first suggested fix in the list above, but can be implemented on your side.

I'll leave the issue open to acknowledge the problem, but we are not very likely to get around looking into more complicated fixes very soon I'm afraid.