kyamagu
SkRuntimeEffect Support
Is your feature request related to a problem? Please describe.
There doesn't appear to be any easy way to apply custom shaders, other than combinations of existing ones. Thus custom sksl shaders are not possible without resorting to native code.
Describe the solution you'd like
Language binding os SkRuntimeEffect and the methods required to compile a custom SKSL shader and bind textures, values etc to them.
Describe alternatives you've considered
It appears just about possible to apply GLSL via the FBO and your window hosting library, but a huge amount of setup.
Additional context
None
https://api.skia.org/classSkRuntimeEffect.html
Upstream examples:
docs/examples/SkSL_*.cpp
We currently has exactly one reference: https://github.com/kyamagu/skia-python/blob/57e705cccaeaf943bba5d3d561d8d38dda4b4e1b/src/skia/Shader.cpp#L2
Likely one of the shader classes depends on it?
The single line include apparently was a mistake - it is not needed (by current shader class code). That said, it might be needed to fix the rest.
Resources: https://shaders.skia.org/ https://skia.org/docs/user/sksl/
@mountainstorm give #274 a go and see what you think? Currently there is only one example:
https://github.com/HinTak/skia-python-examples/blob/main/SkSL_MinifiedSkSL.py
I think I can port another of the 9 upstream c++, but the other 7 involves nesting runtime effects and needs a bit more additional code.
@kyamagu how do you feel about https://github.com/HinTak/skia-python-examples/blob/05494769d0c962b10d4261ee2f4a8d0a03d356f5/SkSL_MinifiedSkSL.py#L18 ?
I think somewhat the raise RuntimeError should be within skia-python (I.e. if no RuntimeError is raised from skia-python, the routine should just return a valid effect) i.e. my comment in https://github.com/HinTak/skia-python/blob/bce29eaddf2ad016065c9a30ea19c3681516a4f3/src/skia/RuntimeEffect.cpp#L17
@HinTak Right, raising a RuntimeError looks more pythonic. Thanks for the implementation!
I'll move the raise inside skia-python at some point then. There are 6 to do, so it is a bit tedious. Maybe I'll macro-it.
@kyamagu the rest of the runtimeeffect codes looks a bit complicated; one issue is argument being passed of the form SkSpan<childptr> which is almost vector<childptr> but not quite. I probably will have to look at how pybind11 convert between c++ vector and python arrays. The other is dictionary-like right assignment: doing a.b['c'] = d where a.b['c'] is a pointer looked up from within a by name, rather than numeric index. That probably mean we have to make that internal look-up works like python dictionaries. The header files only tell you half of the story - I didn't expect the return value (as a pointer) is to be assigned to, on the left-hand side - as an lvalue.
Just pushed a bit more out, now it does 4 of the 9 examples : https://github.com/HinTak/skia-python-examples?tab=readme-ov-file#sksl-examples
@kyamagu I think the rest of the examples needs two things: (1) constructing an array of our types and passing it on as arguments: https://github.com/HinTak/skia-c-examples/blob/a35645b71e6ee3ea6a372a8ac619196920640c31/SkSL/SkSL_EvaluatingNestedShaders.cpp#L28 https://github.com/HinTak/skia-c-examples/blob/a35645b71e6ee3ea6a372a8ac619196920640c31/SkSL/SkSL_EvaluatingNestedShaders.cpp#L36 (2) dictionary-like lvalues: https://github.com/HinTak/skia-c-examples/blob/a35645b71e6ee3ea6a372a8ac619196920640c31/SkSL/SkSL_Uniforms.cpp#L28 . Any clues, pointers on those? I think (1) I need to receive a python list, and do the cast inside pybind11. (2) not sure.
I have managed (1), and thinking about (2). The nested shader python examples are already out, but I an hoping to simplify some of the (1) before adding the changes to the pull.
The (1) turns out to be slightly more complicated - c++ does implicit casts and imageShader as a SkShader is casted to SkRuntimeEffect::ChildPtr in the array, and when the array is passed to makeShader, it is casted to SkSpan on entry.
I have explicit casts in the python code at the moment but would like to hide them, like the corresponding c++ code.
@HinTak Let me try to understand your thoughts.
For 1), you're trying to be able to do the following on the python side, and the children get passed to the makerShader method is the concern, right? In pybind11, it is possible not to implicitly convert a Python object into C++ container and instead just get a py::list object. If std::vector has some trouble with conversion to SkSpan<childptr>, I would suggest doing custom conversion logic to construct SkSpan not from std::vector but directly from the py::list object. Alternatively, it is probably possible to bind SkSpan as an opaque type.
def make_gradient_shader():
# ...
def draw(canvas, image):
image_shader = image.makeShader(
samplingOptions=skia.SamplingOptions(skia.FilterMode.kLinear))
sksl =("""
uniform shader input_1;
uniform shader input_2;
half4 main(float2 coord) {
return input_1.eval(coord) * input_2.eval(coord);
}
""")
children = [image_shader, make_gradient_shader()]
effect = skia.RuntimeEffect.MakeForShader(sksl)
my_shader = effect.makeShader(None, children)
# ...
For 2), it looks like the C++ object has a proxy class to implement a dictionary-like assignment. If you really intend to implement the same behavior, perhaps the option is to implement a proxy class as a pybind11 class (and as a static attribute of the builder), then define a __setitem__ method on it. Otherwise it is much simpler to change the builder method to accept both key and values: e.g., builder.uniform("not_a_color", [1, 0, 0, 1]).
@kyamagu thanks for the notes. I have gotten most of the implicit conversion working - it needs a VectorRuntimeChildPtr as an opaque type: https://github.com/HinTak/skia-python-examples/blob/50e51f13c0bb5353c1398b46b7bea99e55007654/SkSL_EvaluatingNestedShaders.py#L39 (and the corresponding code is already in the pull). I have tried the `py::list route too, but it seems that that I haven't quite got the details right. As far as I see the c++ code looks simple but it does about 3 implicit conversions - from individual shader elements to runtimechildptr packed into an array, then the array on entry pack into span. To achieve this in python, one needs to go from list of shades to list of runtime childptr, then to array of runtimeptr, then to span of runtimeptr. (Span is a vector like template that doesn't own or has a backing storage, so there needs to be an intermediate array / vector somewhere, especially as we receive a python list). I have only managed two of the 3 implicit conversion so far.
I have come to conclusion about the (2) as you do too, doing a __setitem__ (on the proxy class), or have a setuniform("key", value). I am leaning on the former, thought the latter is easier. It needs a lot of return policy stuff and reference type return for this to work too.
Here are my current thoughts/notes on this elsewhere https://github.com/pybind/pybind11/discussions/5423
I think it needs a std::vector somewhere in the middle, as we we need a dynamic array. The c code does a static array allocated and initialized in one step, which also acts as the backing storage of Span, but we need a contiguous storage between span and py::list, so there is not avoiding its use, though it may not needs to be exposed directly as an opaque type. Exposing it as a opaque type is what I found which works at the moment, because it forces implicit conversion before it and after it.
I have concluded that the dictionary access isn't easy - the main problem is that the upstream code uses an overloaded assignment operator (it overloads operator=); and it is not only overloaded but also templated, and that's one fundamental thing one cannot do in python. So actually I needed to do multiple setUniform(string key, type1 value), type2 value), etc. This is also non-API, so probably not worth spending more time on it. Anyway, it seems I have managed to port all the existing upstream examples. (Still to check and push, not yet public).
SkiaSharp (c# binding) seems to have it in a dictionary-like construct, but we'll just have to conclude it isn't easy...
The current state of #274 should be able to run all 9 examples of the python ports: https://github.com/HinTak/skia-python-examples
I have to adding some missing Image and ColorSpace APIs too.
Still a bit rough; main thing is I want to move the runtime error throw from the client to within skia-python.
I have moved the runtime error throwing inside skia-python - so you always get a valid effect compiled, or see a python exception (instead of having to check effect being non-null yourself, then look at the error message on the client side). And simplify my examples, errors don't need to be checked any more.
I.e. If you insert a typo in the SkSL code, you get an exception about where your typo is - instead of in the c++ code, typos gives you a null result and you have to read the errText itself in these circumstances.