Cross-platform `WebView` Implementation with off-screen rendering and OpenGL texture upload

[DexerBR]

Description:

In the context of planning for Kivy 3.0, it would be amazing to implement a cross-platform WebView widget that follows the following workflow:

1. Render the WebView off-screen.
2. Capture the page drawing to a pixel buffer.
3. Upload this pixel buffer to an OpenGL texture, enabling its integration with the Kivy graphical interface.

---

The proposal includes specific approaches for each supported platform:

Desktop (Windows/Linux/macOS): Off-screen rendering will be achieved using the CEF (Chromium Embedded Framework) https://github.com/chromiumembedded/cef. Pixel buffer output is officially supported by the OnPaint method of the CefRenderHandler class, which simplifies integration.

Android: For Android, the approach involves using the native WebView (https://developer.android.com/reference/android/webkit/WebView), configured for off-screen rendering. The best approach for the workflow above needs further investigation.

iOS: For iOS, it is necessary to determine the best strategy for capturing the content of a WebView and rendering it to an OpenGL texture. Considerations include:

• What is the most effective approach for iOS? Are there any recommended frameworks or techniques for off-screen rendering with WebView on iOS?
• Are there frameworks comparable to CEF that can be used on iOS for this purpose?
• What specific challenges might arise when implementing this functionality on iOS?

Additional Context:

The goal is to ensure that, regardless of the platform, Kivy 3.0 can provide a consistent and integrated user experience when rendering web content into OpenGL textures.

Contributions to discuss and define the best approach for Android and iOS are welcome, aiming to ensure a robust and efficient implementation.

---

EDIT 1: Here is a proof of concept for the latest version of CEF (using the most recent stable version, Chromium version 128.0.6613.18) running based on the workflow mentioned above, on Windows. However, the behavior should be identical on both macOS and Linux.

https://github.com/user-attachments/assets/aeeed281-e141-4a8f-95be-4066bc4229a5

[abedhammoud]

that would be amazing!!

[Samael-TLB]

Hi @DexerBR , I was working on something similar. A way to enable native widgets/views to be embedded as proper kivy widgets, which can occupy space in kivy layout, be applied kivy transformations, opacity, clipping as well as handling of native touch events.

Do you want to collaborate and discuss on this further?

[DexerBR]

Hi @DexerBR , I was working on something similar. A way to enable native widgets/views to be embedded as proper kivy widgets, which can occupy space in kivy layout, be applied kivy transformations, opacity, clipping as well as handling of native touch events.

Do you want to collaborate and discuss on this further?

This seems interesting. But first, I am particularly interested in having this working for webviews on iOS/Android, in the same way it works for Windows/MacOS/Linux. The challenge itself would be finding an ideal approach to achieve offscreen rendering on iOS/Android.

[T-Dynamos]

in the same way

CEF on android?

For android we can override draw method of WebView to render to opengl texture.

Example: http://stackoverflow.com/questions/12499396/is-it-possible-to-render-an-android-view-to-an-opengl-fbo-or-texture

[Sahil-pixel]

If CEF support on Android it will be great. And if

in the same way

CEF on android?

For android we can override draw method of WebView to render to opengl texture.

Example: http://stackoverflow.com/questions/12499396/is-it-possible-to-render-an-android-view-to-an-opengl-fbo-or-texture

Ok but is it possible to detect input from mouse & keyboard ? By render to OpenGL texture

[Samael-TLB]

Hi @DexerBR , I was working on something similar. A way to enable native widgets/views to be embedded as proper kivy widgets, which can occupy space in kivy layout, be applied kivy transformations, opacity, clipping as well as handling of native touch events. Do you want to collaborate and discuss on this further?

This seems interesting. But first, I am particularly interested in having this working for webviews on iOS/Android, in the same way it works for Windows/MacOS/Linux. The challenge itself would be finding an ideal approach to achieve offscreen rendering on iOS/Android.

Yes lets start on this then.

Ok but is it possible to detect input from mouse & keyboard ? By render to OpenGL texture

Ideally, you would need to create a virtual widgets tree for android then translate the touches from texture region to appropriate position in that virtual widget hirarchy. The texture is taken from the virtual widget tree. Normally the performance bottleneck is due to the native widgets being rendered into the texture and then using the texture again for rendering in kivy in android, which is something to take hold of. If we can directly render to kivy screen buffer through hardware egl locks it will be better. Under the hood this is what android does for rendering to surface textures to be used in gl.

[DexerBR]

in the same way

CEF on android?

Unfortunately, CEF is not available for Android/iOS. There is an open issue about it, but no updates: https://github.com/chromiumembedded/cef/issues/1991

For android we can override draw method of WebView to render to opengl texture.

Example: http://stackoverflow.com/questions/12499396/is-it-possible-to-render-an-android-view-to-an-opengl-fbo-or-texture

Indeed, I think that overriding OnDraw might be one way to achieve this, but I'm still not sure if there is a way to render directly to the OpenGL texture, without having to compute the pixel buffer and then upload it to the texture.

I did a small draft test in a project in Android Studio and found that:

• The OnDraw method of the webview is not called as the page is drawn when the webview is set to off-screen, so we need to call the texture update logic manually (at a periodic interval).
• There are some scenarios where performance may drop, such as when playing YouTube videos. Furthermore, the videos being played on YouTube do not seem to be captured correctly. Maybe I didn't configure something correctly.

The logic for capturing the pixel buffer was:

public static class CustomWebView extends WebView {
...

private Bitmap bitmap;
private Canvas bitmapCanvas;
...

bitmapCanvas.save();
bitmapCanvas.translate(-scrollX, -scrollY);
this.draw(bitmapCanvas);
bitmapCanvas.restore();

int[] pixels = new int[width * height];
bitmap.getPixels(pixels, 0, width, 0, 0, width, height);

byte[] pixelBytes = convertPixelsToRgbaBytes(pixels);

sendBitmapData(pixelBytes);  // -> glTexSubImage2D

In some scenarios the performance was very good, but in others like YouTube, it should be much better. I think part of this must have to do with the resource consumption in the extra computation in the code above, perhaps the ideal would be to look for a more direct way of rendering to an OpenGL texture that uploading the pixel buffer using glTexSubImage2D

[DexerBR]

Ideally, you would need to create a virtual widgets tree for android then translate the touches from texture region to appropriate position in that virtual widget hirarchy. The texture is taken from the virtual widget tree. Normally the performance bottleneck is due to the native widgets being rendered into the texture and then using the texture again for rendering in kivy in android, which is something to take hold of. If we can directly render to kivy screen buffer through hardware egl locks it will be better. Under the hood this is what android does for rendering to surface textures to be used in gl.

@Samael-TLB Do you have any idea how rendering could be done from an android view to an OpenGL texture, without the need to compute the canvas/bitmap pixels of the Android view and then upload the buffer to the OpenGL texture?

[DexerBR]

Interesting topic here: https://stackoverflow.com/questions/16604150/embedded-chromium-or-webkit-in-android-app

[Sahil-pixel]

Please . I am very excited to see kivy's webview widget. So we can add in kivy's layout. Please initiate quickly. 😃😃😃😃😃

[Sahil-pixel]

Ideally, you would need to create a virtual widgets tree for android then translate the touches from texture region to appropriate position in that virtual widget hirarchy. The texture is taken from the virtual widget tree. Normally the performance bottleneck is due to the native widgets being rendered into the texture and then using the texture again for rendering in kivy in android, which is something to take hold of. If we can directly render to kivy screen buffer through hardware egl locks it will be better. Under the hood this is what android does for rendering to surface textures to be used in gl.

@Samael-TLB Do you have any idea how rendering could be done from an android view to an OpenGL texture, without the need to compute the canvas/bitmap pixels of the Android view and then upload the buffer to the OpenGL texture?

https://github.com/kivy/kivy/blob/master/kivy/core/camera/camera_android.py

https://github.com/Android-for-Python/Camera4Kivy/blob/main/src/camera4kivy/preview_camerax.py

See this

[T-Dynamos]

We could try building and patching CEF but the problem is that it takes 20hrs+ to build.

[Sahil-pixel]

We could try building and patching CEF but the problem is that it takes 20hrs+ to build.

Any idea why it is taking so much time ?. And how you are building cef for Android. ? I want to know.

[Sahil-pixel]

Interesting topic here: https://stackoverflow.com/questions/16604150/embedded-chromium-or-webkit-in-android-app

See Cef Complied for arm https://cef-builds.spotifycdn.com/index.html#linuxarm64

[tito]

Please note about the kivy/cef integration done 10 years ago: https://github.com/kivy-garden/garden.cefpython (and another one: https://github.com/rentouch/cefkivy)

It would be nice to have somebody refreshing the garden and maintaining it.

[DexerBR]

Please note about the kivy/cef integration done 10 years ago: https://github.com/kivy-garden/garden.cefpython (and another one: https://github.com/rentouch/cefkivy)

It would be nice to have somebody refreshing the garden and maintaining it.

Thanks, there are probably some very interesting logics for handling input events.

[Sahil-pixel]

Hi @DexerBR , I was working on something similar. A way to enable native widgets/views to be embedded as proper kivy widgets, which can occupy space in kivy layout, be applied kivy transformations, opacity, clipping as well as handling of native touch events.

Do you want to collaborate and discuss on this further?

Please upload code .so we can see it. Or come to kivy discord and discuss more.

[Sahil-pixel]

It may be helpful. https://stackoverflow.com/questions/44304105/unable-to-render-an-android-webview-into-an-external-texture-shared-between-c

[DexerBR]

It may be helpful. https://stackoverflow.com/questions/44304105/unable-to-render-an-android-webview-into-an-external-texture-shared-between-c

Cool! I believe this is exactly what we're looking for (direct rendering to the texture without needing to copy the pixel buffer via the CPU, allowing everything to be processed directly on the GPU). This should significantly improve performance!

[Sahil-pixel]

I have done some experiment with kivy for Android OS . I have created java ANDROID bitmap from webview and make Texture for kivy and paste texture on Widget of kivy . so it is showing frames of webview in some interval of time .

from kivy.app import App
from kivy.uix.label import Label
from kivy.uix.floatlayout import FloatLayout
from kivy.utils import platform
from kivy.core.image import Image as CoreImage
from kivy.properties import ObjectProperty
from kivy.clock import Clock,mainthread
import io

from android.runnable import run_on_ui_thread
@run_on_ui_thread
def byte_array_to_texture(buf):
	#image = PImage.open()
	#print(image)
	#print(byte_array)
	#for i in byte_array:
	#	print('hello')
	#buf=bytes(byte_array)
	#print(buf)
	im=CoreImage(io.BytesIO(buf), ext='png')
	return im.texture


class MyLayout(FloatLayout):
	_tex=ObjectProperty(None)
	#
	def call(self,):
		if platform=='android':
			from android_webview import WebView
			url='https://www.google.com/'
			self.web=WebView(url,callback=self._call)
			self.web.enable_javascript=True
			self.web._init()
			#self._update(0)
			Clock.schedule_interval(self._update,1/10)
		

		else:
			print('It is Linux')
	@mainthread
	def _call(self,arg):
		print('hello',arg)
		if arg:
			self._tex=arg

	def _update(self,dt):

		self.web.draw()
		#print('Texture######',imtx)
		#self._tex=imtx




class MyApp(App):
	def build(self):
		return MyLayout()

if __name__=="__main__":
	MyApp().run()

#kv file
<MyLayout>:
	canvas.before:
		Color:
			rgba:[1,1,1,1]
		Rectangle:
			pos:self.pos
			size:self.size
	Widget:
		pos_hint: {'center_x': 0.5,'center_y':0.35}
		size_hint: (None, None)
		size:dp(300),dp(300)
		canvas:
			Color:
				rgba:[1,1,1,1]
			Rectangle:
				size:self.size
				pos:self.pos
				texture:root._tex
		
	Button:
		pos_hint: {'center_x': 0.5,'y':0.1}
		size_hint: (None, None)
		size:dp(80),dp(50)
		text: 'press'
		on_release:root.call()

#android_webview.py

# Android **only** HTML viewer, always full screen.
#
# Back button or gesture has the usual browser behavior, except for the final
# back event which returns the UI to the view before the browser was opened.
#
# Base Class:  https://kivy.org/doc/stable/api-kivy.uix.modalview.html 
#
# Requires: android.permissions = INTERNET
# Uses:     orientation = landscape, portrait, or all
# Arguments:
# url               : required string,  https://   file:// (content://  ?) 
# enable_javascript : optional boolean, defaults False 
# enable_downloads  : optional boolean, defaults False 
# enable_zoom       : optional boolean, defaults False 
#
# Downloads are delivered to app storage see downloads_directory() below.
#
# Tested on api=27 and api=30
# 
# Note:
#    For api>27   http://  gives net::ERR_CLEARTEXT_NOT_PERMITTED 
#    This is Android implemented behavior.
#
# Source https://github.com/Android-for-Python/Webview-Example

from kivy.uix.modalview import ModalView
from kivy.graphics.texture import Texture
from kivy.clock import Clock,mainthread
from functools import partial
from android.runnable import run_on_ui_thread
from jnius import autoclass, cast, PythonJavaClass, java_method

WebViewA = autoclass('android.webkit.WebView')
WebViewClient = autoclass('android.webkit.WebViewClient')
LayoutParams = autoclass('android.view.ViewGroup$LayoutParams')
LinearLayout = autoclass('android.widget.LinearLayout')
KeyEvent = autoclass('android.view.KeyEvent')
ViewGroup = autoclass('android.view.ViewGroup')
DownloadManager = autoclass('android.app.DownloadManager')
DownloadManagerRequest = autoclass('android.app.DownloadManager$Request')
Uri = autoclass('android.net.Uri')
Environment = autoclass('android.os.Environment')
Context = autoclass('android.content.Context')
PythonActivity = autoclass('org.kivy.android.PythonActivity')
Canvas=autoclass("android.graphics.Canvas")
Bitmap=autoclass('android.graphics.Bitmap')
BitmapConfig = autoclass("android.graphics.Bitmap$Config")
ByteArrayOutputStream=autoclass('java.io.ByteArrayOutputStream')
CompressFormat = autoclass("android.graphics.Bitmap$CompressFormat")
BitmapUtil = autoclass('org.kivy.player.BitmapUtil')
class DownloadListener(PythonJavaClass):
    #https://stackoverflow.com/questions/10069050/download-file-inside-webview
    __javacontext__ = 'app'
    __javainterfaces__ = ['android/webkit/DownloadListener']

    @java_method('(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;J)V')
    def onDownloadStart(self, url, userAgent, contentDisposition, mimetype,
                        contentLength):
        mActivity = PythonActivity.mActivity 
        context =  mActivity.getApplicationContext()
        visibility = DownloadManagerRequest.VISIBILITY_VISIBLE_NOTIFY_COMPLETED
        dir_type = Environment.DIRECTORY_DOWNLOADS
        uri = Uri.parse(url)
        filepath = uri.getLastPathSegment()
        request = DownloadManagerRequest(uri)
        request.setNotificationVisibility(visibility)
        request.setDestinationInExternalFilesDir(context,dir_type, filepath)
        dm = cast(DownloadManager,
                  mActivity.getSystemService(Context.DOWNLOAD_SERVICE))
        dm.enqueue(request)


class KeyListener(PythonJavaClass):
    __javacontext__ = 'app'
    __javainterfaces__ = ['android/view/View$OnKeyListener']

    def __init__(self, listener):
        super().__init__()
        self.listener = listener

    @java_method('(Landroid/view/View;ILandroid/view/KeyEvent;)Z')
    def onKey(self, v, key_code, event):
        if event.getAction() == KeyEvent.ACTION_DOWN and\
           key_code == KeyEvent.KEYCODE_BACK: 
            return self.listener()
        

class WebView:
    # https://developer.android.com/reference/android/webkit/WebView
    texture=None
    def __init__(self, url, enable_javascript = False, enable_downloads = False,
                 enable_zoom = False,width=800,height=700,callback=None,**kwargs):
        super().__init__(**kwargs)
        self.url = url
        self.enable_javascript = enable_javascript
        self.enable_downloads = enable_downloads
        self.enable_zoom = enable_zoom
        self.webview = None
        self.enable_dismiss = True
        self.width=width
        self.height=height
        self.layout=None
        self.callback=callback
        self.texture=None
        #self._init()

    @run_on_ui_thread     
    def _init(self):
        mActivity = PythonActivity.mActivity 
        webview = WebViewA(mActivity)
        webview.setWebViewClient(WebViewClient())
        webview.getSettings().setJavaScriptEnabled(self.enable_javascript)
        webview.getSettings().setBuiltInZoomControls(self.enable_zoom)
        webview.getSettings().setDisplayZoomControls(False)
        webview.getSettings().setAllowFileAccess(True) #default False api>29
        layout = LinearLayout(mActivity)
        layout.setOrientation(LinearLayout.VERTICAL)
        layout.addView(webview, self.width, self.height)
        mActivity.addContentView(layout, LayoutParams(-1,-1))
        webview.setOnKeyListener(KeyListener(self._back_pressed))
        if self.enable_downloads:
            webview.setDownloadListener(DownloadListener())
        self.webview = webview
        self.layout = layout
        try:
            webview.loadUrl(self.url)
        except Exception as e:            
            print('Webview.on_open(): ' + str(e))
            self.dismiss()  
        
            
    def _dismiss(self):
        if self.enable_dismiss:
            self.enable_dismiss = False
            #parent = cast(ViewGroup, self.layout.getParent())
            #if parent is not None: parent.removeView(self.layout)
            self.webview.clearHistory()
            self.webview.clearCache(True)
            self.webview.clearFormData()
            self.webview.destroy()
            self.layout = None
            self.webview = None
    @run_on_ui_thread
    def draw(self,):
        #self.on_size('', '')
        bitmap=Bitmap.createBitmap(self.width, self.height, BitmapConfig.ARGB_8888)
        canvas = Canvas(bitmap)
        print(self.layout)
        self.layout.draw(canvas)
        print(bitmap)
        
        #stream = ByteArrayOutputStream()
        #bitmap.compress(CompressFormat.PNG, 90, stream)
        size = (bitmap.getWidth(), bitmap.getHeight())
        pixels = bytes(BitmapUtil().toPixels(bitmap))
        print("pixel,size=",size)
        self.texture_making(pixels,size)
    @mainthread
    def texture_making(self,pixels,size):
        #if not self.texture:
        self.texture = Texture.create(size, colorfmt='rgba')
        self.texture.flip_vertical()
        self.texture.blit_buffer(pixels, colorfmt='rgba', bufferfmt='ubyte')
        #print(texture)
        
        Clock.schedule_once(partial(self._callback,self.texture),)
        self.texture=None
        return 
    
    def _callback(self,arg,dt):
        print('@@@@@@',arg)
        self.callback(arg)


        
    @run_on_ui_thread
    def on_size(self, instance, size):
        if self.webview:
            params = self.webview.getLayoutParams()
            params.width = self.width
            params.height = self.height
            self.webview.setLayoutParams(params)

    def pause(self):
        if self.webview:
            self.webview.pauseTimers()
            self.webview.onPause()

    def resume(self):
        if self.webview:
            self.webview.onResume()       
            self.webview.resumeTimers()

    def downloads_directory(self):
        # e.g. Android/data/org.test.myapp/files/Download
        dir_type = Environment.DIRECTORY_DOWNLOADS
        context =  PythonActivity.mActivity.getApplicationContext()
        directory = context.getExternalFilesDir(dir_type)
        return str(directory.getPath())

    def _back_pressed(self):
        if self.webview.canGoBack():
            self.webview.goBack()
        else:
            self.dismiss()  
        return True

BitmapUtil = autoclass('org.kivy.player.BitmapUtil') https://github.com/Android-for-Python/music_service_example/blob/main/mediastore_utils.py#L175-L184

Perfromance see this video https://cdn.discordapp.com/attachments/614483622409535489/1284150070710898719/Record_2024-09-13-19-21-03.mp4?ex=66e59573&is=66e443f3&hm=be69806423f0961c209b7875a6299846223fa4b39074b5e50b553c78e3c89ce5&

https://cdn.discordapp.com/attachments/614483622409535489/1284198572308430940/Record_2024-09-13-22-08-28_468x1040.mp4?ex=66e5c29e&is=66e4711e&hm=461445124cf516cea4f47151142ff91e041f50c0689d48a5f5ba072d78205ab7&

[Sahil-pixel]

It may be helpful. https://stackoverflow.com/questions/44304105/unable-to-render-an-android-webview-into-an-external-texture-shared-between-c

Cool! I believe this is exactly what we're looking for (direct rendering to the texture without needing to copy the pixel buffer via the CPU, allowing everything to be processed directly on the GPU). This should significantly improve performance! Webview by OpenGL Texture Rendering Done for Kivy Android https://github.com/Sahil-pixel/Webview4Kivy

[DexerBR]

Webview by OpenGL Texture Rendering Done for Kivy Android https://github.com/Sahil-pixel/Webview4Kivy

@Sahil-pixel That's great! I'll take a look soon! 👍

[rcnn-retall]

this is kivy?

[rcnn-retall]

这可能会有所帮助。https://stackoverflow.com/questions/44304105/unable-to-render-an-android-webview-into-an-external-texture-shared-between-c

凉！我相信这正是我们正在寻找的（直接渲染到纹理，而无需通过 CPU 复制像素缓冲区，允许直接在 GPU 上处理所有内容）。这应该会显著提高性能！ Webview by OpenGL 为 Kivy Android https://github.com/Sahil-pixel/Webview4Kivy 完成纹理渲染 Hello, I want to be able to hide the window background but show the controls. Is there anything I can do