Media overlays in the streamer

[qnga]

How media overlays should be implemented in the streamer? The question has first been raised here. There media overlays are introduced as a service provided over HTTP. It's not clear for me whether they should be able to be accessed both in-memory and over HTTP depending on the format. In any case, the fetcher seems to me to be the right place to convert SMIL media overlays to in-memory Synchronized Narration.

[HadrienGardeur]

It mostly depends on the plaform.

On the Web, "media overlays" would only be served using HTTPS:

• a Web App would fetch a RWPM over HTTPS
• through this manifest, a number of links to Synchronized Narration documents are discovered
• the navigator used by this Web App would then fetch over HTTPS the Synchronized Narration document, along with the HTML and audio resources referenced by that document

On mobile, apps that use strictly native code would only rely on HTTP to fetch the HTML and audio resources, everything else would be handled in-memory. But apps that rely on a navigator built entirely in JS may behave exactly like Web Apps and fetch the manifest and Synchronized Narration document using HTTP as well.

[qnga]

On mobile, apps that use strictly native code would only rely on HTTP to fetch the HTML and audio resources, everything else would be handled in-memory.

Everything else doesn't include images and CSS, does it? Why this choice? I understand the idea of passing the manifest in memory, but I didn't think that concerned resources too. It seems to me to be conceptually twisted as it relies on two separate mechanisms for resource handling, and moreover prevents remote resource access for some resource types.

Now, assuming this choice in the Kotlin app, SMIL files may be parsed and their content be added as a kotlin property into the link objects of overlaid Html files (and not into dedicated links). Is that your vision?

I though of still another way: converting SMIL into sync narr format and inject them right into concerned Html files in the fetcher. This is particularly interesting if you plan to use audio element to play sound in synchronized narration. A Js lib might be used to play sound and highlight active text, though controls could probably still be native. I really don't known how you are planning to implement all of this.

[HadrienGardeur]

Everything else doesn't include images and CSS, does it? Why this choice?

They're served over HTTPS as well. In my explanation, I only mentioned the resources that are fetched directly, not the ones that are fetched indirectly as HTML gets rendered (images, CSS, JS and fonts).

Now, assuming this choice in the Kotlin app, SMIL files may be parsed and their content be added as a kotlin property into the link objects of overlaid Html files (and not into dedicated links). Is that your vision?

I though of still another way: converting SMIL into sync narr format and inject them right into concerned Html files in the fetcher. This is particularly interesting if you plan to use audio element to play sound in synchronized narration. A Js lib might be used to play sound and highlight active text, though controls could probably still be native. I really don't known how you are planning to implement all of this.

We'll have to do both and the in-memory model will be based on Synchronized Narration anyway.

[qnga]

So, as far as I understand, in mobile apps:

• SMIL resources are mapped to Links containing the Synchronized Narration data
• <item media-overlay="idref"> is mapped to a Link whose alternates property contains the link corresponding to the idref as described above
• when the navigator or the testapp (I don't know) encounters a link with type "application/vnd.syncnarr+json", it doesn't fetch the resource over HTTP, but directly use Publication.linkWithHref method to retrieve the link containing the synchronization data.

Questions:

• What should be the href of the Synchronized Narration link computed from the SMIL resource? The original SMIL path contains the ".smil" extension, but a different href raises an issue with href unicity.
• How should synchronization data be stored in Link objects? In a synNarr member? In otherProperties?

[qnga]

I think I was again mistaken. NOT

• when the navigator or the testapp (I don't know) encounters a link with type "application/vnd.syncnarr+json", it doesn't fetch the resource over HTTP, but directly use Publication.linkWithHref method to retrieve the link containing the synchronization data.

BUT

• when the navigator or the testapp (I don't know) encounters a Link with type "application/vnd.syncnarr+json", it doesn't fetch the resource over HTTP, but directly use the synchronization data contained in the Link.

Questions remain to be answered.

[qnga]

My last interpretation has the drawback of getting the manifest dirty with Synchronization Narration data right inside. I suggest acting as if Media Overlays would be served over HTTP in the decidated resource with SyncNarr being the native format, more like my first interpreration.

Concretely:

• a Publication object should have a syncNarr collection, as resources and readingOrder. This collection would not to be relevant for the manifest, but just a trick in mobile apps.
• The Epub parser would parse all SMIL files and put synchronisation data into syncNarr collection by means of dedicated Links (or another class) that are not directly contained in alternates but instead referenced with an href, as other resources.
• When the client encounters a link with type application/vnd.syncnarr+json (most likely in alternates, it just looks into syncNarr collection for the href.