Try
A tree (vegetation) transperency shader implementaion request
Hello, I saw your HDR implementation screenshots the other day, and the moment I saw the sunset rays passing around, but not through, the tree tops, I've remembered how beautiful and atmospheric was that effect in The Witcher 3. It would be a huuge boost to a "Gothic+" experience. I hope you would not have to rework the whole lighting system to make it work.
In modern games (and I'm assuming for Witcher3 as well) artist makes a special thickness map, for shader to use. However in Gothic there is no such thing as thickness, we have only 4-bit alpha, what is not enough to make convincing effect. At least I'm not aware of any method that can we can use here.
According to Substance3D's documentation they can create a thickness map by tracing from the mesh surface through the mesh. I guess OG could map each color texture texel's position to world space, shoot a ray from that position in the opposite direction of the face normal and then record the length it takes the ray to travel outside the mesh in the thickness map, reusing the color UVs (EDIT: which may not work, depending on how the meshes are UV mapped), but writing to a new texture. The 4-bit alpha needs to be considered to handle vegetation. I think it would be doable, but more work than the short description implies. It would probably also be a good idea to cache the result on disk as calculating it will be costly.
thickness map by tracing from the mesh surface through the mesh
In that case all vegetation thickness would be zero, since grass/tree-leaves a flat 2-sided polygons?
Well yeah, but a leaf isn't very thick anyway. One could set a minimum depth value in this case. For solid parts of a tree it should work as expected, I think. Unless they intersect with leafs, which will probably be a case to consider, especially when animating trees.
EDIT: I don't have experience with thickness maps, thought some more about it. Alpha can be sampled to get a thickness for the leaf itself, where the ray starts. Then all transparent surfaces along the ray add thickness based on the alpha value at the ray intersection point. The ray stops at the bounding box of the model or any solid surface.