libopenshot icon indicating copy to clipboard operation
libopenshot copied to clipboard

Improved PTS support (presentation time stamps)

Open jonoomph opened this issue 3 years ago • 2 comments

This is a large refactor of how FFmpegReader class utilizes the timestamps, to make better/quicker decisions about audio/video data.

Improvements:

  • Improved PTS offset logic (to offset our streams so that at least one of them starts at position 0.0)
  • Removed missing frame detection (not needed anymore)
  • Use timestamps to reason about missing frames
  • Fixed many async decoding issues (jumbled frames, missing beginning or ending frames)
  • Much improved support for video/audio files with missing chunks/gaps (i.e. missing timestamps)
  • Improved CheckWorkingFrames() method, to determine when working frames are ready to be finalized
  • Improved FPS detection for invalid or missing FPS
  • Fixed a crash (only reproducible on Windows), caused by 2 calls to AddImage() for the same frame

What does all this mean? Video/audio files contain packets. Packets have timestamps to help us keep them in order, and know when to use them. However, packets must be decoded into actual usable data first (compressed bytes => image/audio data). Modern decoders (FFmpeg v3+) utilize threads to decode this data, and handle things asynchronously. This refactor mainly focuses on this async code, to dramatically improve the reliability, speed, and accuracy of the frame data we are accessing with FFmpegReader. OpenShot still had lots of legacy code, which expected the current packet to be processed synchronously from the decoder, causing all sorts of mis-timing events, or flat out missing frames.

Bottom line: This PR will improve the performance for most users, prevent crashes for some users, and improve compatibility with many, many video files.

jonoomph avatar Jul 21 '22 19:07 jonoomph

Codecov Report

Merging #837 (0419f2c) into develop (5776efd) will increase coverage by 1.04%. The diff coverage is 80.15%.

@@             Coverage Diff             @@
##           develop     #837      +/-   ##
===========================================
+ Coverage    48.39%   49.43%   +1.04%     
===========================================
  Files          184      184              
  Lines        17889    15780    -2109     
===========================================
- Hits          8657     7801     -856     
+ Misses        9232     7979    -1253     
Impacted Files Coverage Δ
src/CacheBase.h 75.00% <ø> (ø)
src/CacheMemory.h 0.00% <ø> (ø)
src/FFmpegReader.h 66.66% <ø> (ø)
src/CacheDisk.cpp 82.20% <33.33%> (-2.24%) :arrow_down:
src/Frame.cpp 44.10% <50.00%> (+0.15%) :arrow_up:
src/CacheMemory.cpp 87.01% <71.42%> (+0.26%) :arrow_up:
src/FFmpegReader.cpp 73.68% <82.93%> (+7.48%) :arrow_up:
src/FrameMapper.cpp 94.00% <100.00%> (+0.31%) :arrow_up:
src/QtUtilities.h 80.00% <100.00%> (-3.34%) :arrow_down:
src/Timeline.cpp 43.23% <100.00%> (-1.60%) :arrow_down:
... and 127 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

codecov[bot] avatar Jul 21 '22 19:07 codecov[bot]