Support media resumption with session demo app

[spun]

To support the "Media Resumption" feature introduced in Android 11, we need to "implement a MediaSession callback for onPlay()".

With media3, the MediaLibrarySessionCallback is replacing onPrepareFromMediaId, onPrepareFromSearch, onPrepareFromUri, onPlayFromMediaId, onPlayFromSearch, onPlayFromUri with onSetMediaUri, but the simple onPlay with no input is not included and I can't find an easy way to implement it.

Edit: Is not included in the documentation, but the "Media Resumption" feature is also calling onPrepare() and, after reading through MediaSessionLegacyStub, I was able to "fake" an onPrepare() by catching COMMAND_PREPARE (only dispatched inside onPrepare with no input). I couldn't do the same to fake the onPlay since it uses the COMMAND_PLAY_PAUSE and that command could mean onPlay or onPause, but the onPrepare replacement seems to be enough to support the feature for now.

[marcbaechinger]

I understand that media resumption is still using a legacy MediaControllerCompat to talk to your media session. If this is the case then the system would use something like mediaController.getTransportControls().play() to send the play request.

This corresponds to the documentation of MediaSession.Callback.onPlay() saying: Override to handle requests to begin playback..

If this is the case I would expect that such a legacy call needs to be delegated to the Media3 MediaSession that then would call player.play() or player.setPlayWhenReady(true) on the player with which the media session has been built. I expect that this backwards compatibility is provided by the MediaSessionService (of which the MediaLibraryService is a subclass). If this is not the case this would qualified to be labeled as a bug rather than we need to add a method to the session callback.

Can you clarify whether you found that player.play() or player.setPlayWhenReady(true) is not called when media resumption happens?

[spun]

The play() in player is called correctly, my problem is more about the need to override onPlay() to support the feature. Let me list the calls that "Media Resumption" makes to try to explain what I mean better.

Media Resumption Steps (source)

1. System calls onGetRoot/onGetLibraryRoot with EXTRA_RECENT hint
2. The app returns the root of a media tree that contains the recently played media
3. System calls onLoadChildren/onGetChildren with the Root from step 2 as parentId
4. The app returns the recently played MediaItem

Note: At this point, the system shows the static placeholder notification with the play button. If the user taps on the play button:

6. System calls onGetRoot/onGetLibraryRoot with EXTRA_RECENT hint again
7. System connects to MediaSession and issues a play command to it.

---

At first I thought that the System play call (step 7) would include the mediaId from step 4 and I could use the media3 equivalent to MediaSession.onPlayFromMediaId, but it doesn't, it just issues the play command.

After reading the documentation and the UAMP implementation (I've checked the playWhenReady value and is always true, UAMP doesn't have ACTION_PREPARE as supported action), my assumption was that the System expects that, when the play command is issued (step 7), the app MediaSession.onPlay() will:

• Check that nothing is playing right now (UAMP has this check for free, since it's using the MediaSessionConnector from exoplayer)
• Get the last media played (db, preferences, etc)
• Create a MediaItem
• Prepare it
• Play it

Sorry if it's still not clear. My main doubt was to know where should I retrieve and prepare the MediaItem. The docs mention onPlay() and UAMP seems to do it inside its onPlay() equivalent. That's why I was looking for a direct replacement but maybe there is a better solution that I'm no seeing.

[marcbaechinger]

Thanks for the detailed answer! I think best is to add media resumption to the session demo app (see docs). I renamed the issue accordingly and it's kept as an enhancement. I'm not sure when we get around to do that yet I'm afraid

I can confirm the protocol you are describing with the 3 calls from systemui to the service and the play() command at the end.

MediaSessionCompat.Callback.onPlay is in Media3 Player.play() or Player.setPlayWhenReady(true).

The first two calls to the service are used to created the notification. The service is not started by intent, and when systemui unbinds the service is stopped.

Then after potentially a longer duration the user finds the notification and taps play. Now systemui calls the service again to start the service and call getLibraryRoot with the intention to prepare for playback. The service can load the most recent song again, set the media item and prepare and await 'play()'.

The service can recognize that systemui is the caller by its package name and also gets the isRecent flag to know to prepare the player.

[PaulWoitaschek]

I can't get this working at all and the notification is always unresponsive on Android 13.

Pocketcasts added a workaround here: https://github.com/Automattic/pocket-casts-android/commit/ead4b0c92bfcdc16324dca405b55e6f829118617

Maybe that's a fix that can / should be incorporated into media3? @marcbaechinger

@ashiagr did you investigate into a proper solution after your temporary fix?

[marcbaechinger]

Pocketcasts added a workaround here:

I'm not sure how this is related. As far as I can see the code there is using the legacy androidx.media API.

I meant this works with UAMP in the media3 branch after I implemented it. I can test again when I have some cycles.

[ashiagr]

Hey @PaulWoitaschek 👋

I wasn't able to investigate a proper solution yet.

I upgraded ExoPlayer media library to 2.18.2 which corresponds to Media3 1.0.0-beta03 release but it didn't help.

As a workaround, I used legacy flags with stopForeground(...) that improved notifications responsiveness but it might have introduced side effects.

[PaulWoitaschek]

You are correct @marcbaechinger , with uamp it's working. I'll investigate more, thanks 🙏

If someone else is trying this out, here is a patch to update to rc1 m3.patch

[PaulWoitaschek]

I found the issue! And possibly also a better fix for pocket casts.

The main difference between my implementation and the one of the media3 uamp branch is that I don't have that onTaskRemoved stopSelf implementation.

This was causing a dead notification which is hanging. When clicking on the play button in logcat you can see a DeadObjectException.

To reproduce it, remove these lines in onTaskRemoved:

        releaseMediaSession()
        stopSelf()

• Then start playback in uamp.
• Pause it
• Go to the home screen
• Swipe away the app from the recents
• Now the notfication is dead and seems half broken (only a progressbar without time)

Additionally a fix for pocketcasts might be to call stopSelf from onTaskRemoved

[ashiagr]

a fix for pocketcasts might be to call stopSelf from onTaskRemoved

Already tried this @PaulWoitaschek, it doesn't seem to be working for us. Thanks so much for taking a look. 🙏 @marcbaechinger, is there anything you can suggest or things we should be checking to make it work with ExoPlayer's latest version?

[marcbaechinger]

The dead notification is a know issue I think that is caused by the app not stopping the service properly. See this comment also.

is there anything you can suggest or things we should be checking to make it work with ExoPlayer's latest version?

Disclaimer: I see that the commit in PocketCast is for a MediaBrowserService. This may be a bit different, although I think similar constraints apply.

I'm not sure if calling stopSelf in onTaskRemoved is sufficient in all cases. There may be some things preventing the service from being stopped when simply calling stopSelf which would have the effect of onDestroy not being called as expected which may mess things up (see #167).

A controller that is still connected to the session is such a case. In the cases when the controller has connected by using a service connection (which is probably the most common case when building a controller with a token that points to the service component), the controller has bound to the service. Calling stopSelf when a client is bound to the service prevents the service from being stopped I think. If I'm not mistaken, this is why UAMP is calling releaseMediaSession() also. This is to make sure all controllers are unbound when stopSelf is called and the service is actually killed.

When I put log statements in some of the involved components in the session demo app, I see that MediaSessionService.onUnbind() is called before onTaskRemoved is being called. So specific to that session demo app I know that all controllers that have bound to the service have been released (and hence called onUnbind() on the service). In this state of the app calling stopSelf() is sufficient, works as expected and terminate the service properly.

[PaulWoitaschek]

I can confirm that! Initially when studying the UAMP implementation I also thought: What is @marcbaechinger doing there, that's totally unnecessary to tear down the stuff there because onDestroy will be called anyways. So I tried that and it didn't work. Then I added the tearDown in onTaskRemoved as well and it started working 😁

https://github.com/PaulWoitaschek/Voice/blob/c0d2feb3b39efe39a44232fca8c0062c775db65a/playback/src/main/kotlin/voice/playback/session/PlaybackService.kt#L34

[WSteverink]

Hello, I'm currently using a MediaLibraryService and implementing MediaResumption based on the provided documentation Uamp and this topic. However, I'm encountering two issues. Here are some notes related to the implementation:

onGetLibraryRoot In the onGetLibraryRoot method, if params.isRecent is true, it returns a "recent root" MediaItem. Additionally, I'm preparing the player with items to resume if params.isRecent is true AND the player's timeline is empty.

onGetChildren The onGetChildren method returns the stored media items if the parentId matches the ID of the item mentioned above.

onTaskRemoved / onDestroy In the Service, we store the current mediaItems in onTaskRemoved and release the mediaSession afterwards. We also call stopSelf() to clear the Notification.

Issue 1: The problem I've noticed is that onGetChildren with the recentRoot is only called the first time I load something into the player, rather than when the system wants to create a notification for MediaResumption.

I expected this behaviour to occur when the notification is triggered. It's puzzling because the MediaItems we're returning will quickly become outdated as we load other items.

Issue 2: Despite the slight confusion from the first issue, MediaResumption seems to be working as expected.

However, after rebooting the device and selecting play on the Notification, I receive the following error: Reason: Context.startForegroundService() did not then call Service.startForeground():.

It's worth mentioning that I'm not using a Bluetooth controller and I'm testing this on a Pixel 3 device running Android 12.

[marcbaechinger]

I'm currently using a MediaLibraryService and implementing MediaResumption based on the provided documentation Uamp and this topic.

This documentation is outdated I'm afraid. Please accept my apologies.

Unfortunately, we haven't yet published the documentation for the Media3 session module. And yes, I share your sentiments about this :( . We will take action now to improve this situation (see docs below).

Issue 1: The problem I've noticed is that onGetChildren with the recentRoot is only called the first time I load something

That's how it works. This needs to be understood as that System UI asking your app whether you support playback resumption. It will call you once when you post the first notification and you can return whatever you want but it must be at least one playable item (see here).

You only need to return real data during boot time. Please see the documentation below. With Media3 you don't have to implement this library service workflow with System UI yourself. Media3 is doing this for you. You only have to follow the two steps below.

rather than when the system wants to create a notification for MediaResumption.

The system knows the last item that you played because this is reflected in the session at the time the last notification was posted by the app. No need to call you again - all information available.

Despite the slight confusion from the first issue, MediaResumption seems to be working as expected.

Indeed.

However, after rebooting the device and selecting play on the Notification, I receive the following error: Reason: Context.startForegroundService() did not then call Service.startForeground():.

With 1.1.0-rc01 you can override MediaSession.Callback.onPlaybackResumption.

/**
     * Returns the last recent playlist of the player with which the player should be prepared when
     * playback resumption from a media button receiver or the System UI notification is requested.
     *
     * @param mediaSession The media session for which playback resumption is requested.
     * @param controller The controller that requests the playback resumption. This is a short
     *     living controller created only for issuing a play command for resuming playback.
     * @return The {@linkplain MediaItemsWithStartPosition playlist} to resume playback with.
     */
    @UnstableApi
    default ListenableFuture<MediaItemsWithStartPosition> onPlaybackResumption(
        MediaSession mediaSession, ControllerInfo controller) {
      return Futures.immediateFailedFuture(new UnsupportedOperationException());
    }

We wrote a page about this. I'm not sure whether this ends up on DAC in this form, but it may be helpful for you. The following is possible with 1.1.0-rc01. Please file bugs if not:

Playback resumption with Media3

When using a MediaSession, a media app is advertising media playback to the operating system. Being aware of media playback this way, Android can provide means for a user to resume playback after an app has terminated and even after the device has been restarted.

Media3 provides a MediaButtonReceiver and an API that hides the complexities of registering your app and receiving the attempts to resume from the system.

Add the MediaButtonReceiver

To opt-in to playback resumption, add the Media3 MediaButtonReceiver to your manifest. This tells the system and Media3 that your app supports playback resumption:

<receiver android:name="androidx.media3.session.MediaButtonReceiver"
  android:exported="true">
  <intent-filter>
    <action android:name="android.intent.action.MEDIA_BUTTON" />
  </intent-filter>
</receiver>

Implement Callback.onPlaybackResumption()

Then implement MediaSession.Callback.onPlaybackResumption. Your app is responsible for storing the playlist and the start position at sensible moments and later provide the data to Media3.

When playback resumption is requested by either a Bluetooth device, or the System UI playback resumption notification, Media3 calls Callback.onPlaybackResumption to get the stored playlist, start index and position from the app:

override fun onPlaybackResumption(
  mediaSession: MediaSession,
  controller: ControllerInfo
): ListenableFuture<MediaItemsWithStartPosition> {
  val settable = SettableFuture.create<MediaItemsWithStartPosition>()
  scope.launch { settable.set(restorePlaylist()) }
  return settable
}

An app should aim for completing the ListenableFuture as quickly as possible to improve startup latency. The library then prepares the player and starts playback.

In case you've stored further parameters like the playback speed, the repeat mode or whether the shuffle mode is enabled, onPlaybackResumption is a good place to set these params. Playback then resumes after the future has been completed, with these settings already in place.

Playback resumption turned off by default

By default, meaning when no MediaButtonReceiver is registered in the manifest, playback resumption is turned off.

In this mode, the session receives media button events only while the session is alive and the user can't resume playback once your session is released.

[PaulWoitaschek]

I can't get this to working at all.

Here I've implemented what I think should be correct:

https://github.com/PaulWoitaschek/Voice/pull/2012

Tested on Pixel 7, Android 33

Scenario A: Media Button

1. Open Voice
2. Play a Book
3. Pause
4. Kill the app (swipe away from recents)
5. Wait a moment for the process to die
6. Send PlayPause adb shell input keyevent 85
7. onPlaybackResumption is never called
8. The App ANR's

Scenario B: Notification

1. Open Voice
2. Play a Book
3. Pause
4. Kill the app (swipe away from recents)
5. Wait a moment for the process to die
6. Drag down the notification and click on play
7. Nothing happens
8. Click again on play
9. Remove the Logcat filters and observe:

2023-06-28 23:24:43.085 MediaSessionRecord                E  Remote failure in play.
                                                             android.os.DeadObjectException
                                                             	at android.os.BinderProxy.transactNative(Native Method)
                                                             	at android.os.BinderProxy.transact(BinderProxy.java:584)
                                                             	at android.media.session.ISessionCallback$Stub$Proxy.onPlay(ISessionCallback.java:729)
                                                             	at com.android.server.media.MediaSessionRecord$SessionCb.play(MediaSessionRecord.java:1243)
                                                             	at com.android.server.media.MediaSessionRecord$ControllerStub.play(MediaSessionRecord.java:1578)
                                                             	at android.media.session.ISessionController$Stub.onTransact(ISessionController.java:523)
                                                             	at android.os.Binder.execTransactInternal(Binder.java:1280)
                                                             	at android.os.Binder.execTransact(Binder.java:1244)

[WSteverink]

By default, meaning when no MediaButtonReceiver is registered in the manifest, playback resumption is turned off.

In this mode, the session receives media button events only while the session is alive and the user can't resume playback once your session is released.

Thank you for your quick and clear response! I appreciate it. I will definitely try using the onPlaybackResumption callback. Could you please confirm if this callback replaces the current setup we have? In other words, will we no longer need a root media item or get calls to onGetChildren?

If that's the case, I would be delighted to implement it, as I believe it would greatly enhance the code's readability and descriptiveness.

[marcbaechinger]

In other words, will we no longer need a root media item or get calls to onGetChildren?

Yes. Media3 is doing a root media item for recent items for you under the hood. When you have added the MediaButtonReceiver and System UI is calling onGetLibraryRoot, Media3 will respond.

When System UI calls onGetChildren(recentRootMediaId) for the given recent root item, Media3 will respond as well. In this case there are two variants:

• Player is in STATE_IDLE: This is the case when System UI asks for the recent item during the boot process of the device. In this case Media3 calls Callback.onPlaybackResumption() to get the metadata that is required by System UI to create the notification.

• Player is not in STATE_IDLE: In this case, Media3 (by default) has already posted a notification at this very moment. So the purpose of the call from System UI is simply to know whether the app actually wants to support playback resumption with a notification. Media3 responds with the smallest possible response to make this work which is a playable item.

Could you please confirm if this callback replaces the current setup we have?

Yes, this is the intention. The intention is that an app for supporting playback resumption with Media3 has to do two things:

1. Add the MediaButtonReceiver provided by Media3 to the AndroidManifest.xml. With this an app opts-in to playback resumption and also needs to do do the next point.
2. Implement Callback.onPlaybackResumption and return the items with which the player should be prepared when playback resumption is requested.

All implementation details are done by Media3. If this is not the case, please file a bug.

I will implement this in UAMP as soon as possible. I hope I can provide this very soon as part of the media3 branch of UAMP or a PR of this branch.

[PaulWoitaschek]

Do you think I have a flaw in my implementation or do you think there is a bug in media3?

[WSteverink]

Do you think I have a flaw in my implementation or do you think there is a bug in media3?

I think what you have added to the Manifest differs from the snippet above. I have just tested the new approach with 1.1.0-rc01 and it seems to work just fine.

[marcbaechinger]

@PaulWoitaschek

Yeah, not sure, sorry. Can you try to remove

<action android:name="android.intent.action.MEDIA_BUTTON" />

from the service and have it only in the receiver? That's a long shot though. For Media3 it shouldn't make a difference, just to remove to be sure no one else sees this. The other actions your having declared for the service are sufficient for the Media3 library to see your service.

Can you when the app is running and after terminating the app, do

adb shell dumpsys media_session | grep MediaButtonReceiver

?

If the MediaButtonReceiver from the manifest is in place, then you should see the registered media button receiver being a broadcastIntent.

Last MediaButtonReceiver: MBR {pi=PendingIntent{a036b8c: PendingIntentRecord{881a2d5 androidx.media3.demo.session broadcastIntent}}, type=1}

Without a receiver in the manifest Media3 registers the service as the receiver during the life-time of the session, but clears it out after release. In this case the the Last MediaButtonReceiver is a foreground service intent

Last MediaButtonReceiver: MBR {pi=PendingIntent{3bcf6a5: PendingIntentRecord{3397a7a androidx.media3.demo.session startForegroundService}}, type=3}

when the app is terminated this is cleared out, because playback resumption is not supported. The Last MediaButtonrecevier should then be null in the output.

[PaulWoitaschek]

It makes no difference if I remove that.

After swiping away the service, it says:

Last MediaButtonReceiver: MBR {pi=PendingIntent{e4ddf21: PendingIntentRecord{5aa3f46 de.ph1b.audiobook broadcastIntent}}, type=1}

I added some logging and what looks suspicious is this:

VoicePlayer:194                   D  setPlayWhenReady=true, playbackState=4

Apparently from `Util.java this function is called

public static boolean handlePlayButtonAction(@Nullable Player player) {
    if (player == null) {
      return false;
    }
    @Player.State int state = player.getPlaybackState();
    boolean methodTriggered = false;
    if (state == Player.STATE_IDLE && player.isCommandAvailable(COMMAND_PREPARE)) {
      player.prepare();
      methodTriggered = true;
    } else if (state == Player.STATE_ENDED
        && player.isCommandAvailable(COMMAND_SEEK_TO_DEFAULT_POSITION)) {
      player.seekToDefaultPosition();
      methodTriggered = true;
    }
    if (player.isCommandAvailable(COMMAND_PLAY_PAUSE)) {
      player.play();
      methodTriggered = true;
    }
    return methodTriggered;
  }

Which then does nothing because no media item was set yet.

This is the call stack when the prepare() is being called.

Which again looks weird because no playback resumption seems involved here.

If I set media items in prepare, the media buttons work. But that's probably not intended, is it?

[PaulWoitaschek]

Ha, we found out in the very same moment that it's related to play pause (85). If I send 126 (play), the resumption is working as expected.

[marcbaechinger]

Thanks for the stack trace!

From that it looks like it doesn't go the expected code path. But I think it goes a path that Media3 should avoid when playback resumption is requested by Bluetooth with a KEYCODE_MEDIA_PLAY_PAUSE instead of KEYCODE_MEDIA_PLAY. We can just rewrite the intent to PLAY.

Thanks for reporting! I filed #493

If you want to verify, you can copy the MediaButtonReceiver into your project: https://github.com/androidx/media/blob/main/libraries/session/src/main/java/androidx/media3/session/MediaButtonReceiver.java I would be happy if you can give it a try. I was thinking this is determined by the system but seems it's the BT device? The three headsets I have seem to send a PLAY then.

Then add the following on line 140, the last line of the if branch if (Util.SDK_INT >= 26) {:

intent
          .getExtras()
          .putParcelable(
              Intent.EXTRA_KEY_EVENT,
              new KeyEvent(keyEvent.getAction(), KeyEvent.KEYCODE_MEDIA_PLAY));

If you now use your own MediaButtonReceiver the fix makes sure a KEY_EVENT_PLAY is sent to the service and playback resumption should be triggered.

[PaulWoitaschek]

I tried that but it doesn't have an effect. I also tried to copy more infos from the KeyEvent.

I am actually not sure if this is a real world issue of if this is not just an artificial scenarios that only might be seen by users who use stuff like tasker etc.

[WSteverink]

.....

1. Add the MediaButtonReceiver provided by Media3 to the AndroidManifest.xml. With this an app opts-in to playback resumption and also needs to do do the next point.
2. Implement Callback.onPlaybackResumption and return the items with which the player should be prepared when playback resumption is requested.

All implementation details are done by Media3. If this is not the case, please file a bug.

I will implement this in UAMP as soon as possible. I hope I can provide this very soon as part of the media3 branch of UAMP or a PR of this branch.

Small question corresponding the new implementation. Is there a way to optionally allow media resumption? Lets say i only want to support and show the Notification if the user is logged in.

Previously we where able to do this in onGetLibraryRoot.

[PaulWoitaschek]

Try to subclass the receiver and only conditionally call super.onReceive

[marcbaechinger]

Try to subclass the receiver and only conditionally call super.onReceive

The receiver is only called when playback resumption is requested by a Bluetooth head set. The System UI resumption notification is starting the service directly.

Small question corresponding the new implementation. Is there a way to optionally allow media resumption? Lets say i only want to support and show the Notification if the user is logged in.

This is currently not possible I'm afraid. Would you want to allow BT resumption but not the notification or wouldn't this be to be separated?

[WSteverink]

Try to subclass the receiver and only conditionally call super.onReceive

The receiver is only called when playback resumption is requested by a Bluetooth head set. The System UI resumption notification is starting the service directly.

Small question corresponding the new implementation. Is there a way to optionally allow media resumption? Lets say i only want to support and show the Notification if the user is logged in.

This is currently not possible I'm afraid. Would you want to allow BT resumption but not the notification or wouldn't this be to be separated?

I don't think we need it to be separated. In our use case it would be pretty straight forward, we just want to allow our users to play our content while logged in with a valid account.

Meaning that if we store a list of media items to resume before a user logs out we don't allow him to continue playing afterwards. We could just fix it with what we return in onPlaybackResumption, but the prettiest solution would be to hide the media notification :) .

[PaulWoitaschek]

For my use case, I also need to prevent playback resumption conditionally.

For instance if the user removed his audiobook, there is nothing I can play when a user requests resumption.

[marcbaechinger]

For instance if the user removed his audiobook, there is nothing I can play when a user requests resumption.

Once you tell System UI you want a notification when playback starts, you can't get back and change that decision. System UI doesn't give you a second chance. This would work for BT resumption but as far as I can tell not with the System UI notification.

How did you implement this with the old API and the System UI notification?

[PaulWoitaschek]

I don't think I handled it correctly 🙈

The BT resumption dismissal could be fixed by subclassing the receiver and intercepting onReceives already right now if I understand correctly.