hzwer
Artefacts described in this ticket are improved in SVP with Rife v4.15 but still present
When using SVP, Rife tends to have less artefacts than the other mode. I notice that Rife 4.7 has even less artefacts than 4.6. But the problem is that the artefacts that remain tend to be worse. The biggest one I've seen recently is a character appearing and disappering every other frame. Is there any use for me to point out the location of these artefacts ? Is there anyone I should send them to?
Hello,
I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors.
If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live.
But yeah, you could add your videos probably to here and we could if those errors still appear.
Hello,
I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors.
If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live.
But yeah, you could add your videos probably to here and we could if those errors still appear.
Thanks. I separate SVP errors from Rife errors. As you said, SVP errors with Rife tend to happen on scene changes. But these are not the same as Rife errors which happen less often but generally look much worse. Rife 4.7 uses about 20% more GPU memory for me, but it has improved things with one major artefact gone and another has been reduced.
Right now, there is one horrible error where the character literally blinks in and out each alternate frame. I have attached a video showing the issue. You can't miss it.
https://github.com/hzwer/Practical-RIFE/assets/140253706/756ebfe3-84cf-42ce-8cea-bd402660855f
@netExtra I'd say, the best thing is to test out https://nmkd.itch.io/flowframes.
If you test out all different models and all of them exhibit this error, then this might be a rife problem.
It probably is, because most of the rife problems are very fast motion problems, where the ai has to guess, what happens.
BTW: You can also test out all other methods beside rife very easily and if this still happens, then this might even be an univerisal problem that cannot be solved probably.
@netExtra I'd say, the best thing is to test out https://nmkd.itch.io/flowframes.
If you test out all different models and all of them exhibit this error, then this might be a rife problem.
It probably is, because most of the rife problems are very fast motion problems, where the ai has to guess, what happens.
BTW: You can also test out all other methods beside rife very easily and if this still happens, then this might even be an univerisal problem that cannot be solved probably.
I've tested 4.6 and 4.7 , both have the same issue and since 4.7 > 4.6 > 4.4 I'm guessing that 4.4 will be the same. Maybe I'm missing something here but even the pay version of Flowframes seems to be only using 4.4 which has even more artefacts and errors.
FlowFrames Pay Version uses up to 4.6. But I'd say, try out RIFE 3/2/1 in FlowFrames Free versions and perhaps they are not present there.
FlowFrames Pay Version uses up to 4.6. But I'd say, try out RIFE 3/2/1 in FlowFrames Free versions and perhaps they are not present there.
Installed Flowframes. Says GPU: Virtual Desktop Monitor even though I have an Nvidia GPU. Tried changing GPU id makes no difference. When interpolating CPU is basically idling for scene extraction and frame extraction. If it's going to be this slow I really would need a realtime mode.
@netExtra OK if all of them have it, then it's probably RIFE Model, probably needs more training.
Hello,
I think these kind of errors you can report here, but be aware, that SVP will probably never be fully able make it able to play smoothly and without errors.
If you have a video and there are scene changes. Then everybody needs a good strategy how to handle it, but how you want to handle it while playing a video live.
But yeah, you could add your videos probably to here and we could if those errors still appear.
Thanks. I separate SVP errors from Rife errors. As you said, SVP errors with Rife tend to happen on scene changes. But these are not the same as Rife errors which happen less often but generally look much worse. Rife 4.7 uses about 20% more GPU memory for me, but it has improved things with one major artefact gone and another has been reduced.
Right now, there is one horrible error where the character literally blinks in and out each alternate frame. I have attached a video showing the issue. You can't miss it.
https://github.com/hzwer/Practical-RIFE/assets/140253706/756ebfe3-84cf-42ce-8cea-bd402660855f
cases like this are pretty common for large movements and you pretty much can't do anything. there are several algorithms out there that are trained for large movement objects where you can test it out. There is also a paid software called SVFI available on steam, their team has brought out a new feature where you can mask out the objects that are not moving for example crosshair and game map, in your case it would be the character that are barely moving in the centre.
cases like this are pretty common for large movements and you pretty much can't do anything. there are several algorithms out there that are trained for large movement objects where you can test it out. There is also a paid software called SVFI available on steam, their team has brought out a new feature where you can mask out the objects that are not moving for example crosshair and game map, in your case it would be the character that are barely moving in the centre.
Thanks but there were other ugly errors in Rife 4.6 that have been fixed in 4.7/4.8 so I would hope that this eventually gets fixed soon. Especially since these large and ugly errors don't occur in SVP when not using Rife.
4.9 is better. One remainder of an artefact that was improved in 4.7 is now completely gone in 4.9. The Spider-man artefact I show above seems to have improved slightly. I have another similar artefact that has also improved to the point where I can remove it by decreasing the scene detection threshold. It's also a little more efficient than 4.7 too Rife is getting better and better 👍
Update 04/11/2023 I don't know if this helps but I can remove the Spider-man artefact shown above by significantly reducing rife-scene change threshold in SVP. But it is so low that it cause stuttering issues elsewhere.
The 'scene change threshold in SVP' you mentioned is out of rife's control. Because it use a different 'solution' to detect scene changes. Once some frames were marked by this 'solution', rife do nothing on them.
Ideally, choose a clip without scene-changes to test. If you have to use SVP, then totally disable it's scene change algorithm.
Ideally, choose a clip without scene-changes to test. If you have to use SVP, then totally disable it's scene change algorithm.
I disabled the sc algorithm. I double checked it was off by setting the sc threshold to 5 which I know fixes the issue in SVP manager GUI. When I run the clip it's broken just like the video attached above. When I re-enable the algorithm the artefact disappears but causes major problems elsewhere so it's not a solution.
I'm not sure how testing a clip without scene-changes would help because there are no issues with those and never have been.
choose a clip without scene-changes to test.
It's not talking about the movement of len.
Could you give the raw clip of your test src?
choose a clip without scene-changes to test.
It's not talking about the movement of len.
Could you give the raw clip of your test src?
Hi. Sorry I am confused about your request for a clip without scene changes. I don't have any issues with that. What do you mean by "movment of len"?. Also, my raw clip is the one that is attached to this issue. Thanks.
I only see the screen recording here. https://github.com/hzwer/Practical-RIFE/issues/34#issuecomment-1747789930
I want to test its raw clip on my side.
I only see the screen recording here. #34 (comment)
I want to test its raw clip on my side.
OK I'm doing it now. Will upload it in a couple of minutes. Thanks.
Attached. I had to convert it to MP4.
https://github.com/hzwer/Practical-RIFE/assets/140253706/49bbf0e0-cfd1-4957-aa44-3364894a9060
Thanks. It could be reproduced by inference_video.py with model4.9.
https://github.com/hzwer/Practical-RIFE/assets/41094733/6414e4cd-a169-47bd-a23a-d67244210687
Thanks. It could be reproduced by
inference_video.pywith model4.9. https://github.com/hzwer/Practical-RIFE/assets/41094733/6414e4cd-a169-47bd-a23a-d67244210687
Now it can be replicated does that help? BTW the attached video doesn't work :)
BTW the attached video doesn't work
It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec.
I didn't try legacy v3/2 models so I have no ideas more.
BTW the attached video doesn't work
It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec.
I didn't try legacy v3/2 models so I have no ideas more.
OK Thanks. I've raised a new issue for Rife 4.11 but I'm not sure if I should be since it probably doesn't affect the model training.
OK so this artefact seems to be gone now and I am confused because up until this morning it was still there with Rife 4.9. I have another annoying artefact similar to this which was getting better but seems to have regressed a little. SVP has been updated and new code added so I will have to investigate further.
Update: The problem artefact is back again and the one that had regressed is back to being improved again. Very strange! Not going to reopen at the moment.
OK there was a weird issue where the player I was using wasn't registering with SVP after the first file. I'm guessing that is why the artefacts went away because SVP and therefore RIFE was not running.
BTW the attached video doesn't work
It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec.
I didn't try legacy v3/2 models so I have no ideas more.
I don't know if this helps, but the issue I sent you the clip of can be significantly reduced in SVP if SC is set below 6.0. The problem is that setting SC this low causes stuttering elsewhere, especially with slow pans
BTW the issue applies to Rife v4.6 as well so it probably goes back some way.
BTW the attached video doesn't work
It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec. I didn't try legacy v3/2 models so I have no ideas more.
I don't know if this helps, but the issue I sent you the clip of can be significantly reduced in SVP if SC is set below 6.0. The problem is that setting SC this low causes stuttering elsewhere, especially with slow pans
BTW the issue applies to Rife v4.6 as well so it probably goes back some way.
At this point you're just confusing SC thresholds. Setting a scene change threshold literally outright disables RIFE on those frames, so obviously there wouldn't be artifacting, and hence why it will obviously be stuttery at some parts.
BTW the attached video doesn't work
It works on my device. You watched it by browser right? unfortunately it seems to be the chromium's issue which doesn't support its codec. I didn't try legacy v3/2 models so I have no ideas more.
I don't know if this helps, but the issue I sent you the clip of can be significantly reduced in SVP if SC is set below 6.0. The problem is that setting SC this low causes stuttering elsewhere, especially with slow pans BTW the issue applies to Rife v4.6 as well so it probably goes back some way.
At this point you're just confusing SC thresholds. Setting a scene change threshold literally outright disables RIFE on those frames, so obviously there wouldn't be artifacting, and hence why it will obviously be stuttery at some parts.
I have disabled the SC threshold in SVP and the artefact is improved but it is still there. But as I said above, I can fix the artefact by setting SVP SC to below 6 but setting SC this low causes stuttering for slow pans.
Here is a a direct quote from the SVP Rife AI FAQ: Scene change threshold - Higher values gives more interpolation artifacts when scene change was not detected. Lower values give more "stuttering" (i.e. repeated frames instead of interpolated) when there was a false detection.
FYI with SC disabled, slow pans are almost perfect but fast object movement causes double images in Rife 4.9 and below. But the double images are gone in Rife 4.12, so there is a visible improvement between 4.9 and 4.12.
In my experience, Rife 4.1 is the best model for panning. 4.12 is slightly worse than 4.1. And 4.13 is slightly worse than 4.12. All other v4 models have significantly more artifacts in fast panning scenes.
