GPUImageView and FFmpeg

[IronSight87]

Hy everybody,

does anybody tried to use GPUImageView with FFmpeg to display the decoded frames and for conversion from yuv to rgb formats?

[hansoong]

Use ffmpeg sws_scale() function can convert yuv to rgba or other format you want

look at https://github.com/lajos/iFrameExtractor

then you will get AvPicture contain raw data of rgba

then create a similar class like gpuImagePicture, with following init method

• (id)initWithRawData:(AVPicture *)pict width:(int)width height:(int)height smoothlyScaleOutput:(BOOL)smoothlyScaleOutput { if (!(self = [super init])) { return nil; } self.shouldSmoothlyScaleOutput = smoothlyScaleOutput; [GPUImageOpenGLESContext useImageProcessingContext];

  _sizeOfImage = CGSizeMake(width, height);

  // For now, deal with images larger than the maximum texture size by resizing to be within that limit CGSize scaledImageSizeToFitOnGPU = [GPUImageOpenGLESContext sizeThatFitsWithinATextureForSize:_sizeOfImage]; if (!CGSizeEqualToSize(scaledImageSizeToFitOnGPU, _sizeOfImage)) { _sizeOfImage = scaledImageSizeToFitOnGPU; }

  if (self.shouldSmoothlyScaleOutput) { // In order to use mipmaps, you need to provide power-of-two textures, so convert to the next largest power of two and stretch to fill CGFloat powerClosestToWidth = ceil(log2(_sizeOfImage.width)); CGFloat powerClosestToHeight = ceil(log2(_sizeOfImage.height));

  _sizeOfImage = CGSizeMake(pow(2.0, powerClosestToWidth), pow(2.0, powerClosestToHeight));
  

  }

  CFDataRef data = CFDataCreateWithBytesNoCopy(kCFAllocatorDefault, pict->data[0], pict->linesize[0]*height,kCFAllocatorNull);

  GLubyte *imageData = NULL; imageData = (GLubyte *) CFDataGetBytePtr(data);

  glBindTexture(GL_TEXTURE_2D, outputTexture); if (self.shouldSmoothlyScaleOutput) { glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR_MIPMAP_LINEAR); }

  glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, (int)_sizeOfImage.width, (int)_sizeOfImage.height, 0, GL_RGBA, GL_UNSIGNED_BYTE, imageData);

  if (self.shouldSmoothlyScaleOutput) { glGenerateMipmap(GL_TEXTURE_2D); } CFRelease(data);
  return self; }

then check gpuImage Example how to display it with gpuImageView

[IronSight87]

Thank you for your answer. I already tried this, but sws_scale consums a lot of performance and i get an cpu load of 140%. Is there something I have to pay attension, when I compile ffmpeg (neon, asm ...)?

[hansoong]

Compile for armv7

./configure
--disable-asm
--enable-gpl
--enable-postproc
--enable-swscale
--enable-avfilter
--prefix=armv7
--enable-cross-compile --target-os=darwin --arch=arm --cpu=cortex-a8 --enable-pic
--arch=arm
--target-os=darwin
--cc=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/gcc
--as="gas-preprocessor.pl /Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/gcc"
--sysroot=/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS5.1.sdk
--extra-cflags="-w -arch armv7 -mfpu=neon "
--extra-ldflags="-arch armv7 -mfpu=neon -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS5.1.sdk"

create SwsContext for RGBA format

img_convert_ctx = sws_getCachedContext(NULL, pCodecCtx->width, pCodecCtx->height, pCodecCtx->pix_fmt, outputWidth, outputHeight, PIX_FORMAT, SWS_FAST_BILINEAR, NULL, NULL, NULL);

scale it

sws_scale (img_convert_ctx, pFrame->data, pFrame->linesize, 0, pCodecCtx->height, picture.data, picture.linesize);

hope this can help, I never check cpu performance but work quite smooth for me on ipod 4.

[IronSight87]

Okay, thanks I will try this. Have you ever tried to convert yuv to rgb (bgra) frames with shaders On the GPU? I tried to modify the glcameraripple example, But I have not been able to get ist working ;)

[hansoong]

never try this example before

[garayronald]

Did you find out how to convert yuv to rgb? I too am getting CPU usage of +100% using swscale and would like to decrease that.

[IronSight87]

Sadly, No

[progamertr]

Brad, is this feature planning in near future? ( I want to use ffmpeg and gpuimage together)

[tqk2811]

Did you find out how to convert yuv to rgb? I too am getting CPU usage of +100% using swscale and would like to decrease that.

@garayronald, @IronSight87 are you found solution? (such as use gpu).
i using sws_scale got x2 cpu than scrcpy, and scrcpy source code only send raw frame to sdl