KhronosGroup
Remaining issues with the tangent space
A gist of open questions and suggestions wrt glTF tangent space definitions (mostly sourced from #1252).
-
The MikkTSpace algorithm is not directly applicable to an indexed mesh that was not exported with tangents in mind. Such meshes require "unindexing" before processing.
- [x] Suggested action: None, since it's an implementation detail.
-
The MikkTSpace algorithm has one configurable parameter (
fAngularThreshold). It is not used by default.- [x] Suggested action: None. This is not an issue since the spec explicitly mentions "... default MikkTSpace ...".
-
The key to correctly working normal maps is that exactly the same process is used both for baking and for sampling per-fragment normal vector values. This comprises two sub-issues:
-
Per-vertex vs per-fragment bitangent reconstruction. A misalignment there may break lighting in certain cases. It seems that the industry is moving towards the per-fragment approach.
- [ ] Suggested action: TBD.
-
At which point(s) should renormalization occur?
- [ ] Suggested action: TBD.
-
Note that the third entry of the list above applies to all normal maps, even when the tangents are provided.
References:
- http://www.mikktspace.com/
- https://github.com/mmikk/MikkTSpace/blob/master/mikktspace.h
- https://bgolus.medium.com/generating-perfect-normal-maps-for-unity-f929e673fc57
/cc @bghgary @emackey @javagl @donmccurdy
Per-vertex vs per-fragment bitangent reconstruction
We probably want to base this on what the authoring tools are doing... they don't tend to be open source though, do we know what's more common?
There's also: https://github.com/KhronosGroup/glTF-Sample-Models/issues/174#issuecomment-779532934, about doing a .w *= -1 flip — compared to using MikkTSpace output directly — to account for the texture coordinate system.
about doing a
.w *= -1
This works to flip the handedness. Another way to look at this same problem is to take 1.0 - texcoord_0.y as the V coordinate input to the MikkTSpace algorithm, as it has a different "vertical" sense from glTF, and this causes all clockwise / counter-clockwise windings in UV space to be flipped.
More references on the derivative-based approach:
-
Bump Mapping Unparametrized Surfaces on the GPU (Morten S. Mikkelsen, 2010)
- Follow-up: https://mmikkelsen3d.blogspot.com/2011/07/derivative-maps.html
- Explainer: https://www.rorydriscoll.com/2012/01/11/derivative-maps/
- Surface Gradient–Based Bump Mapping Framework (Morten S. Mikkelsen, 2020)
Per-vertex vs per-fragment bitangent reconstruction
We probably want to base this on what the authoring tools are doing... they don't tend to be open source though, do we know what's more common?
Authoring tools that leverage the standard MikkT convention expect to reconstruct the bitangent in the fragment shader, saving valuable interpolant registers that would increase your LDS footprint, reducing occupancy. While it may appear that the computational burden is higher doing this per-pixel, this is highly geometry dependent. Consider all the occluded and backfacing vertices that would otherwise be shaded but ultimately discarded before rasterization. Note also that interpolation and renormalization afterwards is somewhat inexact anyways, since the arc taken by any unit vector interpolated this way will not have constant angular velocity (this is an nlerp, not a slerp). The other consideration is content with non-unit tangents/bitangents needed for tiling detail maps with skew/stretch. These lengths can be encoded in the 4-component "MikkT" tangent, decoded in VS, and then reapplied in the PS after local-to-world normals are computed.
I think it is covered by the "At which point(s) should renormalization occur?" point but I wanted to explicitly highlight a situation that I find unclear what to do when trying to adhere to the mikktspace way.
When non-uniform (or even uniform...) scaling is used, the vertex tangents loaded from a model should be transformed by the model matrix, but this will change their scale. Given the vector cross product of two vectors A x B = ||A|| * ||B|| * sin(angle between) * normal, unless the tangent is first renormalised before reconstructing the bitangent, then the bitangent too will be scaled up by the length of the tangent. Then when multiplying the TBN with the tangent-space normal, the tangent space normal will be stretched along the tangent and bitangent basis vectors.
I don't know when is going to be the correct point at which to renormalise, but it would seem like perhaps renormalising the tangent vector after transforming by the model matrix would make sense?
The MikkTSpace algorithm is not directly applicable to an indexed mesh that was not exported with tangents in mind. Such meshes require "unindexing" before processing.
* [x] **Suggested action**: None, since it's an implementation detail.
Noobs like myself would appreciate it if this were at least documented in the spec. The spec currently says:
"When tangents are not specified, client implementations SHOULD calculate tangents using default MikkTSpace algorithms with the specified vertex positions, normals, and texture coordinates associated with the normal texture."
It was not entirely clear to me when I SHOULD calculate the tangents, and after spending several hours debugging, I realized MikkTSpace is not supposed to be invoked on meshes with vertex indices, at least not without unindexing the mesh first. The DamagedHelmet sample seems like a good example of this; I get a very nice seam along the +Z axis that bleeds into the lighting.
Also, variability like this makes specs harder to implement. I would have thought at first that for simplicity and load speed, if a mesh were normal-mapped and the author intended that tangent vectors be used, the spec would have required the author to export the tangents as opposed to require the client to post-process the mesh. The former would keep the client lean and mean. The tangents would also make the glTF file heavier, but I take it those vectors compress fairly well.
Per-vertex vs per-fragment bitangent reconstruction
We probably want to base this on what the authoring tools are doing... they don't tend to be open source though, do we know what's more common?
Authoring tools that leverage the standard MikkT convention expect to reconstruct the bitangent in the fragment shader, saving valuable interpolant registers that would increase your LDS footprint, reducing occupancy.
That would be my hunch too. The MikkTSpace documentation appears to be a bit ambiguous about whether to compute the bitangent in a vertex or fragment shader:
// Should you choose to reconstruct the bitangent in the pixel shader instead
// of the vertex shader, as explained earlier, then be sure to do this in the normal map sampler also.
One other difference I found with the glTF viewer reference implementation (vert, frag) is that the reference implementation normalizes vectors several times over. The MikkTSpace documentation on the other hand appears to want you to use unnormalized/interpolated vectors all the way and only normalize at the end:
// To avoid visual errors (distortions/unwanted hard edges in lighting), when using sampled normal maps, the
// normal map sampler must use the exact inverse of the pixel shader transformation.
// The most efficient transformation we can possibly do in the pixel shader is
// achieved by using, directly, the "unnormalized" interpolated tangent, bitangent and vertex normal: vT, vB and vN.
// pixel shader (fast transform out)
// vNout = normalize( vNt.x * vT + vNt.y * vB + vNt.z * vN );
// where vNt is the tangent space normal. The normal map sampler must likewise use the
// interpolated and "unnormalized" tangent, bitangent and vertex normal to be compliant with the pixel shader.
So I think there is a bit of a mismatch between the reference implementation and what MikkTSpace is asking. I think it would also help if the glTF spec explained how to compute the bitangent in shaders in a bit more detail, or at least leave a short note with a link to the reference implementation.
I have had an email conversation with Morten Mikkelsen about my problem of applying scaling (uniform or non-uniform) to a normal-mapped model and so where to apply normalisation and not. The takeaways are as follows:
- the priorities are:
- do the exact opposite process of the normal map baker to avoid introducing errors
- do things in the vertex stage if possible to avoid unnecessary per-fragment computation
- as such, the mikktspace algorithm avoids normalisation in the fragment shader - do not renormalise the interpolated tangent space
- in the vertex stage:
- transform the vertex normal by the inverse transpose local to world matrix and then normalise it
- transform the xyz components of the vertex tangent by the local to world matrix and normalise it. Preserve the w component (I think, I’ll double check this)
Then either in the vertex stage or fragment stage (but importantly at the same point as is done when baking!), calculate the bitangent using the cross product of the world normal and world tangent xyz, multiplied by the world tangent w.
One more issue that needs to be considered is that if there is a local to world matrix applied after the vertex tangents were calculated (i.e. in a glTF transform hierarchy or even in some subsequent transform hierarchy after the model is loaded into an application) then sign of the determinant of this local to world matrix must be multiplied with the vertex tangent w component (the bitangent sign).
I've been testing scaling models like flight helmet with negative scales and stumbled across some lighting bugs that were fixed by the above, which was pointed out to me by Morten Mikkelsen. They pointed me to section 2 of https://jcgt.org/published/0009/03/04/ but I didn't manage to understand exactly why as I was unable to understand the geometric meaning of matrix N.
The hopefully final problem that I face in getting correct lighting when applying a negative scale to FlightHelmet is to do with the hose being double-sided and in that case I need to correctly handle flipping the normals. The original author of the code I'm fixing also flipped the tangent and bitangent and I can't seem to find any references for this being correct.