BradLarson
[MovieInput]A Video decode issue
My app uses GPUImage2 to processing video files. I found that some video files can not be decoded correctly with MovieInput. The decoded images look like this:
And here is the video file. The person in the video is not myself :-) video.mp4.zip
I tried the Objective-C version GPUImage. Using GPUImageMovie to decode the same video file and the images are decoded correctly
After comparing the code with the Objective-C version, I found some differences here (MovieInput.swift: line 160 to 170):
let luminanceFramebuffer = sharedImageProcessingContext.framebufferCache.requestFramebufferWithProperties(orientation:.portrait, size:GLSize(width:GLint(bufferWidth), height:GLint(bufferHeight)), textureOnly:true)
luminanceFramebuffer.lock()
glActiveTexture(GLenum(GL_TEXTURE0))
glBindTexture(GLenum(GL_TEXTURE_2D), luminanceFramebuffer.texture)
glTexImage2D(GLenum(GL_TEXTURE_2D), 0, GL_LUMINANCE, GLsizei(bufferWidth), GLsizei(bufferHeight), 0, GLenum(GL_LUMINANCE), GLenum(GL_UNSIGNED_BYTE), CVPixelBufferGetBaseAddressOfPlane(movieFrame, 0))
let chrominanceFramebuffer = sharedImageProcessingContext.framebufferCache.requestFramebufferWithProperties(orientation:.portrait, size:GLSize(width:GLint(bufferWidth), height:GLint(bufferHeight)), textureOnly:true)
chrominanceFramebuffer.lock()
glActiveTexture(GLenum(GL_TEXTURE1))
glBindTexture(GLenum(GL_TEXTURE_2D), chrominanceFramebuffer.texture)
glTexImage2D(GLenum(GL_TEXTURE_2D), 0, GL_LUMINANCE_ALPHA, GLsizei(bufferWidth / 2), GLsizei(bufferHeight / 2), 0, GLenum(GL_LUMINANCE_ALPHA), GLenum(GL_UNSIGNED_BYTE), CVPixelBufferGetBaseAddressOfPlane(movieFrame, 1))
and the Objective-C version is (GPUImageMovie.m: line 526 to 562):
if ([GPUImageContext deviceSupportsRedTextures])
{
err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, [[GPUImageContext sharedImageProcessingContext] coreVideoTextureCache], movieFrame, NULL, GL_TEXTURE_2D, GL_LUMINANCE, bufferWidth, bufferHeight, GL_LUMINANCE, GL_UNSIGNED_BYTE, 0, &luminanceTextureRef);
}
else
{
err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, [[GPUImageContext sharedImageProcessingContext] coreVideoTextureCache], movieFrame, NULL, GL_TEXTURE_2D, GL_LUMINANCE, bufferWidth, bufferHeight, GL_LUMINANCE, GL_UNSIGNED_BYTE, 0, &luminanceTextureRef);
}
if (err)
{
NSLog(@"Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
}
luminanceTexture = CVOpenGLESTextureGetName(luminanceTextureRef);
glBindTexture(GL_TEXTURE_2D, luminanceTexture);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
// UV-plane
glActiveTexture(GL_TEXTURE5);
if ([GPUImageContext deviceSupportsRedTextures])
{
err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, [[GPUImageContext sharedImageProcessingContext] coreVideoTextureCache], movieFrame, NULL, GL_TEXTURE_2D, GL_LUMINANCE_ALPHA, bufferWidth/2, bufferHeight/2, GL_LUMINANCE_ALPHA, GL_UNSIGNED_BYTE, 1, &chrominanceTextureRef);
}
else
{
err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, [[GPUImageContext sharedImageProcessingContext] coreVideoTextureCache], movieFrame, NULL, GL_TEXTURE_2D, GL_LUMINANCE_ALPHA, bufferWidth/2, bufferHeight/2, GL_LUMINANCE_ALPHA, GL_UNSIGNED_BYTE, 1, &chrominanceTextureRef);
}
if (err)
{
NSLog(@"Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
}
chrominanceTexture = CVOpenGLESTextureGetName(chrominanceTextureRef);
glBindTexture(GL_TEXTURE_2D, chrominanceTexture);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
I tried to change the code in Swift to using "CVOpenGLESTextureCacheCreateTextureFromImage" to generating the "luminanceTexture" and "chrominanceTexture", and everything went fine.
Here is the code I have changed:
var luminanceGLTexture: CVOpenGLESTexture?
glActiveTexture(GLenum(GL_TEXTURE0))
let luminanceGLTextureResult = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, sharedImageProcessingContext.coreVideoTextureCache, movieFrame, nil, GLenum(GL_TEXTURE_2D), GL_LUMINANCE, GLsizei(bufferWidth), GLsizei(bufferHeight), GLenum(GL_LUMINANCE), GLenum(GL_UNSIGNED_BYTE), 0, &luminanceGLTexture)
assert(luminanceGLTextureResult == kCVReturnSuccess && luminanceGLTexture != nil)
let luminanceTexture = CVOpenGLESTextureGetName(luminanceGLTexture!)
glBindTexture(GLenum(GL_TEXTURE_2D), luminanceTexture)
glTexParameterf(GLenum(GL_TEXTURE_2D), GLenum(GL_TEXTURE_WRAP_S), GLfloat(GL_CLAMP_TO_EDGE));
glTexParameterf(GLenum(GL_TEXTURE_2D), GLenum(GL_TEXTURE_WRAP_T), GLfloat(GL_CLAMP_TO_EDGE));
let luminanceFramebuffer: Framebuffer
do {
luminanceFramebuffer = try Framebuffer(context: sharedImageProcessingContext, orientation: .portrait, size: GLSize(width:GLint(bufferWidth), height:GLint(bufferHeight)), textureOnly: true, overriddenTexture: luminanceTexture)
} catch {
fatalError("Could not create a framebuffer of the size (\(bufferWidth), \(bufferHeight)), error: \(error)")
}
// luminanceFramebuffer.cache = sharedImageProcessingContext.framebufferCache
luminanceFramebuffer.lock()
var chrominanceGLTexture: CVOpenGLESTexture?
glActiveTexture(GLenum(GL_TEXTURE1))
let chrominanceGLTextureResult = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, sharedImageProcessingContext.coreVideoTextureCache, movieFrame, nil, GLenum(GL_TEXTURE_2D), GL_LUMINANCE_ALPHA, GLsizei(bufferWidth / 2), GLsizei(bufferHeight / 2), GLenum(GL_LUMINANCE_ALPHA), GLenum(GL_UNSIGNED_BYTE), 1, &chrominanceGLTexture)
assert(chrominanceGLTextureResult == kCVReturnSuccess && chrominanceGLTexture != nil)
let chrominanceTexture = CVOpenGLESTextureGetName(chrominanceGLTexture!)
glBindTexture(GLenum(GL_TEXTURE_2D), chrominanceTexture)
glTexParameterf(GLenum(GL_TEXTURE_2D), GLenum(GL_TEXTURE_WRAP_S), GLfloat(GL_CLAMP_TO_EDGE));
glTexParameterf(GLenum(GL_TEXTURE_2D), GLenum(GL_TEXTURE_WRAP_T), GLfloat(GL_CLAMP_TO_EDGE));
let chrominanceFramebuffer: Framebuffer
do {
chrominanceFramebuffer = try Framebuffer(context: sharedImageProcessingContext, orientation: .portrait, size: GLSize(width:GLint(bufferWidth), height:GLint(bufferHeight)), textureOnly: true, overriddenTexture: chrominanceTexture)
} catch {
fatalError("Could not create a framebuffer of the size (\(bufferWidth), \(bufferHeight)), error: \(error)")
}
// chrominanceFramebuffer.cache = sharedImageProcessingContext.framebufferCache
chrominanceFramebuffer.lock()
Although this codes decode images correctly, but it is not perfect. Because it allocates two FrameBuffer objects every time when decoding every video frames. I noticed that the "texture" field in FrameBuffer class is a const declared with "let", and the texture is new generated by "CVOpenGLESTextureCacheCreateTextureFromImage" function. So it seems impossible to reuse a exsiting FrameBuffer object from the cache. I think it needs more consideration and discussion here. Anyway, I have commit a pull request with the codes above.
The reason why this issue happening, I thougnt, is same with the issue I have met in Android. Because the wrong-decoded-image is the same. I have raised a question here: https://stackoverflow.com/questions/40541187/transcode-h264-video-by-mediacodec-directly-withou-texture-render In that situation, I tried to display the output-image-data-buffer which are decoded from the decoder, directly. And I got the same wrong images as this time. So I think it may be the same reason. Because the count of bytes in output buffers are not exactly the value of width*height*byteCountInEachPixel. It may have some more byte in each row. And sometimes, the video frame buffer is rotated in 90 or 270 degree, because some camera device will save the video file in a certain landscape ratio, and then add a meta-data of the "rotation" into the video files meta-data area. In that case, a video player should care all these details to display a image from the decoded-output-buffer.So it is a better way to using a OS-provided-API to processing the data from the deocoder's output. Because English is not my mother tongue, I don't know if I have explained clearly about that.Happy to have more communications :-)
Thanks a lot. This saved me. I just copied your changes (as my own version of GPUImage is really far away from original repo so couldn't just pull) without even trying to understand what it changes, but it did saved me :) In my case video was wrong when there was something with changing resolution.
