libass
Extreme lag on heavy "move" elements in dialogue
It seems that when there are move animations in the sub file dialog, it causes the renderer to lag heavily, and in some cases crash the worker. I was previously using version 2.2.0, but saw that there were recently some promising improvements with alternate render techniques lossyRender and blendRender. After trying the latest code with different combinations of both the lossy and blend render modes, I can't seem to get the rendering to improve.
While using the blendRender, I've also tried setting dropAllAnimations to true, and targetFps to 5. While I am no longer seeing the working crashing, the move animations are not dropped, and the animation duration lags past the timeline position where they are supposed to disappear. I've also tried different combinations of setting libassMemoryLimit and libassGlyphLimit both lower (in hopes that it will reduce the resources being used and simply skip over heavy loads), and higher (in hopes that the increased resources would improve render performance). Not sure what is the proper combination of options to get optimal performance from files like the one shown below.
Below is a sample of the dialog combination which is attempting to be rendered. Was wondering if @JustAMan or anyone else possibly had any recommendations on configurations to improve rendering performance, or at least allow the renderer to keep in sync with the timeline and just skip frames when under heavy load.
[Script Info]
Title:
Original Script:
Original Translation:
Original Editing:
Original Timing:
Synch Point:
Script Updated By:
Update Details:
ScriptType: v4.00+
Collisions: Normal
PlayResX: 640
PlayResY: 360
Timer: 0.0000
WrapStyle: 0
[V4+ Styles]
Format: Name,Fontname,Fontsize,PrimaryColour,SecondaryColour,OutlineColour,BackColour,Bold,Italic,Underline,Strikeout,ScaleX,ScaleY,Spacing,Angle,BorderStyle,Outline,Shadow,Alignment,MarginL,MarginR,MarginV,Encoding
Style: Default,Arial,20,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,0,0,0,0,100,100,0,0,1,2,2,2,0010,0010,0010,1
Style: Main,Trebuchet MS,24,&H00FFFFFF,&H000000FF,&H00000000,&H00000000,0,0,0,0,100,100,0,0,1,2,1,2,0010,0010,0018,1
[Events]
Format: Layer,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\move(900,95,535,100)\frz281.1\t(\fs18)\c&H666765&\3c&HE6E7E4\shad0&\frx0\fry354}One One One One One One One
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\an5\c&H696A68&\3c&HECEDEB&\shad0\bord4\fs12\frz23.82\frx18\fry328\move(600,-65,360,55)\t(\fs8)}Two\NTwo
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\an9\c&H2C2D2A&\3c&HECEDEB&\shad0\bord4\fs18\frx18\fry328\move(800,-40,465,80)\t(\fs12)\frz20.62}Three Three Three\NThree Three Three\NThree Three Three\NThree
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\an6\c&H1D1D1D&\3c&HECEDEB&\shad0\bord4\fs16\frz23.82\frx18\fry328\move(224,104,156,195)\t(\fs8)}Four Four\NFour Four Four
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\an6\c&H1D1D1D&\3c&HECEDEB&\shad0\bord4\fs14\frz23.82\frx18\fry328\move(275,68,184,172)\t(\fs7)}Five
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\an4\c&H1D1D1D&\3c&HECEDEB&\shad0\bord4\fs10\frx18\fscx70\fry328\move(132,250,107,274)\t(\fs6)\frz19.87}Six Six Six Six Six Six Six Six Six\Six Six Six Six Six Six Six Six Six,\Nbut Six Six Six Six
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\u1\c&H0E0F0D&\3c&HE4E5E3&\shad0\fs18\an1\move(201,332,150,310)\frx0\fry336\frz15.76\t(\fs8)}Victim{\u0}\NSeven Seven
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\an6\c&H1D1D1D&\3c&HECEDEB&\shad0\bord2\fs16\frz23.82\frx18\fry328\move(412,50,255,163)\t(\fs6)}Eight Eight Eight
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\c&H0E0F0D&\3c&HE4E5E3&\shad0\fs14\an2\move(464,320,282,308)\frz15.76\t(\fs6)\frx6\fry332}Nine\NNine
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\an4\c&H1D1D1D&\3c&HECEDEB&\shad0\bord4\fs14\frx18\fscx70\fry328\t(\fs8)\frz19.87\move(405,170,244,232)}\h\h\h\hTen Ten Ten Ten Ten Ten Ten Ten Ten?\N Ten Ten Ten Ten Ten\N\h\h\h\hTen Ten Ten Ten Ten Ten Ten Ten Ten Ten {right}
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\an5\c&H696A68&\3c&HECEDEB&\shad0\bord4\fs12\frz23.82\frx18\fry328\move(310,-10,196,132)\t(\fs8)}Eleven Eleven\NEleven Eleven Eleven\NEleven Eleven Eleven Eleven\NEleven Eleven Eleven Eleven Eleven Eleven
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\an5\c&H696A68&\3c&HECEDEB&\shad0\bord4\fs12\frz23.82\frx18\fry328\move(460,-45,292,92)\t(\fs8)}Twelve\NTwelve\NTwelve\NTwelve
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\an9\c&H2C2D2A&\3c&HECEDEB&\shad0\bord4\fs18\frx18\fry328\move(800,90,462,172)\t(\fs12)\frz20.62}Thirteen Thirteen Thirteen)\N\N\NThirteen
Dialogue: 0,0:00:00.00,0:00:15.00,Main,Sign,0000,0000,0000,,{\an5\c&H2C2D2A&\3c&HD4D7D9&\shad0\bord2\fs16\frz23.82\frx18\fry328\move(710,240,426,270)\t(\fs8)}Fourteen Fourteen\N\NFourteen Fourteen
Dialogue: 0,0:00:04.50,0:00:08.00,Main,Sign,0000,0000,0000,,okay?
Dialogue: 0,0:00:09.00,0:00:15.00,Main,Sign,0000,0000,0000,,Maybe not okay? Should be gone by now
It is also worth noting that this is specifically being seen when the canvas renderer is hosted within a webview in an Android mobile application.
Thanks in advance!
Was able to improve the performance by reducing the number of times setCurrentTime is called, and lowering the targetFps value. Currently using a canvas, and calling setCurrentTime on a requestAnimationFrame loop, but I guess this is too much for any of the renderers with a move animation.
Using the base renderer still works okay, but still lags and is choppy. The lossy render mode provides improved performance, with less delay on this move animation, but still a little choppy. The blend render mode provided the best performance, but unfortunately when rendering to a canvas and calling setCurrentTime, for some reason it is not able to clear out a rendered text line until there is another line to be rendered. So during times when there should be no text shown, the last rendered line is still visible. This was not the case when using the default renderer or lossy render mode.
I did a test in desktop chrome using the player option, rather than canvas and setCurrentTime, and did not see the same issue with the blend render mode. So for some reason seems to only be happening when using a canvas and calling setCurrentTime.
@JustAMan Have you seen this kind of case when using the blend mode? Would really like to utilize your awesome work on this, but not having rendered lines removed when there should be no text displayed is the last thing preventing me from using it. So using lossy mode for now
Was away for quite a while... I'm not sure I've encountered move elements during my testing.
Also I don't remember upstreaming here the options you're talking about, so I presume you're using our fork (https://github.com/jellyfin/JavascriptSubtitlesOctopus/). If so, what are results with render ahead mode enabled?
@JustAMan I put in a pull-request for your repository, also upon testing render ahead I get visual discrepancies like this:
from what I can tell the animations don't get rendered or get half rendered and end up getting like that. And testing even further, blend render mode keeps the subtitles around longer than needed.
I encountered the issue with low performance and eventual running out of memory on certain intensive subs when supplying a custom canvas and calling setCurrentTime through requestAnimationFrame manually as I was overlaying ass subtitles onto a YouTube iFrame.
Initially I thought it was due to the rate at which requestAnimationFrame was called (120 fps on my monitor) so I forced rAF to only be called at 10 fps. However this led to a bad viewing experience when any kind of animation was used.
Looking around the code I discovered that there is an _isPaused state being handled by subtitles-octopus-worker.js which defaults to true and doesn't change on its own unless a video element is provided.
By calling the octopus instance's setIsPaused whenever the YouTube iFrame played/paused, I was able to get the renderer to perform the same as when a video element is supplied to octopus directly. I also reduced the frame rate of my rAF/setCurrentTime to 4 fps (subtitle-octopus fills the gaps).
When _isPaused is true, only 1 frame is drawn per call to setCurrentTime but it increases when _isPaused is false (and seems to also not trigger the drawing immediately, correct me if I'm wrong).
@nopol10 That worked for me, thank you! I'm working with a lot of karaoke-style animations, and they were very choppy when I used an <audio> element, custom canvas and setCurrentTime but after adding a few of the same event listeners that subtitles-octopus.js adds to videos, it renders beautifully. In particular, it does seem like calling setIsPaused makes a huge difference.
