Add support for video captions

[skjnldsv]

fix https://github.com/nextcloud/viewer/issues/239 (add subtitles for video feature #239)

Generate HTML5 subtitle tracks dynamically based on the contents of the folder where the video is located.

For example: given a movie 'video.mkv', the player will look for VTT subtitle files named 'video.XX.vtt' and '.video.XX.vtt', where XX is the two-letter code for the language.

Signed-off-by: Frederic Ruget [email protected]

[douzebis]

fix #239 (add subtitles for video feature #239)

Generate HTML5 subtitle tracks dynamically based on the contents of the folder where the video is located.

For example: given a movie 'video.mkv', the player will look for VTT subtitle files named 'video.XX.vtt' and '.video.XX.vtt', where XX is the two-letter code for the language.

Signed-off-by: Frederic Ruget [email protected]

Thanks a lot @skjnldsv (and @beardhatcode)!

I now have setup a dedicated test environment with a dedicated server separate from my production instance. Also I have rebuild all from source (server and viewer directly github).

--> I was able to check that the modifications to Videos.vue do not break master.

I hope to make some progress this weekend.

[douzebis]

[ Intermediate status report ]

Hi @skjnldsv and @beardhatcode. I believe I have gathered most of the pieces I will need to implement the requested changes and I am now confident that I will be able to get it to completion.

I have installed xdebug so I can breakpoint/watch the server-side code from vscode, plus I use chrome's builtin javascript debugger for the vue code on client side.

I use apps/forms's ApiController.php and SubmissionService.php for inspiration.

Currently I have three new/modified files:

• apps/viewer/
  • appinfo/routes.php
  • lib/Controller/VideoController.php
  • src/components/Videos.vue

On client side, I use generateUrl() and axios() to call the server, and I package the filename as a query parameter to the url.

On server side, I use OCP\Files\File to get access to the filesystem utility functions, and I use OCP\IL10N to retrieve factory.this.availableLanguages.

So far I have successfully implemented roundtrip communication between client and server, passing parameters to the server and retrieving the response on client side.

[douzebis]

One question still:

From OCP\IL10N I can retrieve a list of abbreviated language names (e.g.: en, fr). However I was not able to retrieve the unabbreviated language names (e.g.: English, French). Is there another variable where this information is present? Or a function I could call from the php controller?

[skjnldsv]

Currently I have three new/modified files:

You can push them, so we can review and comment :)

[skjnldsv]

From OCP\IL10N I can retrieve a list of abbreviated language names (e.g.: en, fr).

You have those in OCP\L10N\IFactory https://nextcloud-server.netlify.app/classes/ocp-l10n-ifactory

[douzebis]

Currently I have three new/modified files:

You can push them, so we can review and comment :)

OK. I'll try and push later today.

[skjnldsv]

OK. I'll try and push later today.

It's sunday, don't pressure yourself! :hugs:

[douzebis]

I have just pushed to feat/video-caption. The code is refactored and working :)

A couple of questions as comments in 'VideoController.php' -- about @nnotations (I don't understand what they do).

Looking forward to studying your review.

[douzebis]

@beardhatcode @skjnldsv I suppose we want some non-regression test to be added to viewer's automated test suite?

Maybe we can setup a mini video folder, have viewer "render" the video and parse the resulting "Elements" (as in chrome -> inspect). E.g.:

• test-video/
  • video.mkv (minimal in size)
  • video.en.vtt (can be empty)
  • .video.fr.vtt (dotted)
  • video.someotherextension

-> Parse the resulting HTML5 for correctness: checking video.en.vtt and .video.fr.vtt are present as tracks and nothing else.

1. Is there a description of NC's test framework?
2. How can we get viewer to render the video page from within the test?
3. What other tests would be needed? (if any)

[douzebis]

@beardhatcode @skjnldsv what about user documentation?

Should the feature be mentioned in the Github's README for the viewer?

[beardhatcode]

@beardhatcode @skjnldsv I suppose we want some non-regression test to be added to viewer's automated test suite?

Yes

Maybe we can setup a mini video folder, have viewer "render" the video and parse the resulting "Elements" (as in chrome -> inspect).

I think we should do this by querying the captions of the plyr object. I think that is the best way to test this.

1. Is there a description of NC's test framework?

It is different for every app, but since you added PHP code, we will also need to add some PHP test (something like this). We'll also need to add video.mkv.spec.js.

2. How can we get viewer to render the video page from within the test?

Yes, see the video.mkv.spec.js

3. What other tests would be needed? (if any)

We should have PHP and cypress tests

[beardhatcode]

Oh and you should also test video's and captions with odd names like:

this 🎥 has 🗨️🗨️🗨️ subtitles🕵️‍♀️ %20.mp4
this 🎥 has 🗨️🗨️🗨️ subtitles🕵️‍♀️ %20.en.vtt

[beardhatcode]

I have been thinking, wouldn't it be better to use the fileid in the API (no escaping issues).

And, with @CarlSchwan search suggestion we can try all the locales :). I was also wondering what:

		$video = $this->rootFolder
			->getUserFolder($this->userSession->getUser()->getUID())

gives on public shares.

I might continue working on this over the next month or so (if not, and you are reading this in 2 months, feel free to continue the work 😅)

[MrRinkana]

I think it would be awesome if you could add support for looking for these .vtt files in hidden directories of the same name as the video file.

I.e. :

• "aVideoName.mp4"
• ".aVideoName" (a hidden folder that contains the "aVideoName.en.vtt" etc.)

I believe that there is benefit to having auxillary files to the videos (captions, thumbnail and such) all be hidden and grouped, the first to not have extra things visible in the files view, the second to not create chaos when people choose to view hidden files to add or change the auxillary files.

Imagine 6 translations per video + thumbnail etc. - unless every video is in its own directory, which I don't think many have, this will be hard to navigate.

I believe supporting this subdirectory approach is the way to go, also for thumbnails (and different qualities if that gets implemented) but that's not relevant to this pull req.