add sws_scale function to upscale the resolution

[0x3375]

Hello. I want to add sws_scale function. So, I made DecoderFFmpeg::scaledResolution() function which is called in DecoderFFmpeg::UpdateVideoFrame.

AVFrame* DecoderFFmpeg::scaleResolution(AVFrame* frame, int scaledWidth, int scaledHeight)
{
	mVideoInfo.isScaled = true;
	mVideoInfo.scaledWidth = scaledWidth;
	mVideoInfo.scaledHeight = scaledHeight;

	SwsContext* pSwsCtx;

	pSwsCtx = sws_getContext(frame->width, frame->height, AV_PIX_FMT_YUV420P,
		scaledWidth, scaledHeight, AV_PIX_FMT_YUV420P,
		SWS_FAST_BILINEAR, NULL, NULL, NULL);

	AVFrame* pScaledFrame = av_frame_alloc(); 
	int ScaledByte = avpicture_get_size(AV_PIX_FMT_YUV420P, scaledWidth, scaledHeight);
	pBuffer = (uint8_t*)av_malloc(ScaledByte * sizeof(uint8_t)); 

	avpicture_fill((AVPicture*)pScaledFrame, pBuffer, AV_PIX_FMT_YUV420P, scaledWidth, scaledHeight); 
	sws_scale(pSwsCtx,
		frame->data, frame->linesize, 0, frame->height,
		pScaledFrame->data, pScaledFrame->linesize); 

	sws_freeContext(pSwsCtx);
	av_frame_free(&frame); 

	return pScaledFrame;
}

void DecoderFFmpeg::updateVideoFrame() 
{
	int isFrameAvailable = 0;
	AVFrame* frame = av_frame_alloc();

	clock_t start = clock();

	if (avcodec_decode_video2(mVideoCodecContext, frame, &isFrameAvailable, &mPacket) < 0)
	{
		LOG("Error processing data. \n");
		return;
	}
	
	LOG("updateVideoFrame = %f\n", (float)(clock() - start) / CLOCKS_PER_SEC);

	if (isFrameAvailable)
	{
		std::lock_guard<std::mutex> lock(mVideoMutex);
		mVideoFrames.push(scaleResolution(frame, 4096, 2048)); 
		updateBufferState();
	}
}

Now, frames in "mVideoFrames queue" have upscaled resolution, so I modified ViveMediaDecoder::DoRendering() function like below.

void DoRendering (int id)
{
	LOG("[DoRendering] %d\n", nDoRender++);
	if (s_DeviceType == kUnityGfxRendererD3D11 && g_D3D11Device != NULL) 
	{
		ID3D11DeviceContext* ctx = NULL;
		g_D3D11Device->GetImmediateContext (&ctx); 
		shared_ptr<VideoContext> localVideoContext;
		if (getVideoContext(id, localVideoContext))  
		{
			AVHandler* localAVHandler = localVideoContext->avhandler.get(); 
			
			if (localAVHandler != NULL && localAVHandler->getDecoderState() >= AVHandler::DecoderState::INITIALIZED && localAVHandler->getVideoInfo().isEnabled) 
			{ 
				if (localVideoContext->textureObj == NULL) // 6
				{
					// unsigned int width = localAVHandler->getVideoInfo().width;
					// unsigned int height = localAVHandler->getVideoInfo().height;
					
					unsigned int width = localAVHandler->getVideoInfo().scaledWidth; // I modified
					unsigned int height = localAVHandler->getVideoInfo().scaledHeight; // I modified

					localVideoContext->textureObj = make_unique<DX11TextureObject>(); 
					localVideoContext->textureObj->create(g_D3D11Device, width, height); 
				}
				double videoDecCurTime = localAVHandler->getVideoInfo().lastTime;
				if (videoDecCurTime <= localVideoContext->progressTime) 
				{
					uint8_t* ptrY = NULL;
					uint8_t* ptrU = NULL;
					uint8_t* ptrV = NULL;
					double curFrameTime = localAVHandler->getVideoFrame(&ptrY, &ptrU, &ptrV); 
					if (	ptrY != NULL && 
							curFrameTime != -1 && 
							localVideoContext->lastUpdateTime != curFrameTime) 
					{
						localVideoContext->textureObj->upload(ptrY, ptrU, ptrV);  // error occured here
						localVideoContext->lastUpdateTime = (float)curFrameTime; 
						localVideoContext->isContentReady = true; 
					}
					localAVHandler->freeVideoFrame();
				}
			}
		}
		ctx->Release();
	}
}

Only "mVideoBuffMax"th frames have been rendered, and a runtime error occurs when trying to render the "mVideoBuffMax+1"th frame.

according to my LOG in localVideoContext->textureObj->upload(),

std::thread YThread = std::thread([&]() {
    if (mWidthY == rowPitchY) {
        LOG("----------------- [Y2] before memcpy\n");
        memcpy(ptrMappedY, ych, mLengthY);  // error occured here
        LOG("----------------- [Y3] after memcpy \n");
    }
}

LOG "[Y3] after memcpy" doesnt called when the current frame number is mVideoBuffMax+1. When I checked the mLengthY, there was no problem. The value of mLengthY was [upscaled width * upscaled height]. why this error happens and how can I fix?

Thank you.

[kyo8568131]

Hi leeseungho1,

I am not sure what the root cause is from the code you post here. But I have some suggestions:

1. Remember to release your pBuffer allocated in scaleResolution.
2. sws_scale is CPU based upscale. One alternative method is to render your frame to a rendertarget with the resolution you wanted.
3. You could attach your native project to Unity for more debug details by step each line. Please check and try again. If you still have some difficulties, please provide more information for further evaluation. Thanks.