scp-fs2open
Add optional deferred lighting pass for cockpits
This adds a flag to enable an additional deferred lighting pass for cockpits. Behind a flag, since this is an option that might cause performance drop in large missions. As the details in cockpits are fairly high and fairly close to the viewer, trying to render the cockpit in world space to avoid needing a second pass for lighting and shadows does not work. This is because at world space distances of several tens of kilometers, the small floating point deviations cause clearly visible vibrations that make rendering in world space unfeasible. Hence the, for now, most sensible solution is to add this second pass, even if it introduces more performance cost.
Just adding here that this visual update really makes cockpits feel substantially more immersive and dynamic.
How did you determine that floating point inaccuracies are the cause for the visual artifacts?
I am a bit confused why the method you are using right now would not also show those artifacts since you are still doing floating point computations within the same ranges.
How did you determine that floating point inaccuracies are the cause for the visual artifacts?
That it's floating point issues is more the conjecture that we still need to be precise 7 to 8 magnitudes below what the operating range of these floats are to not see any issue with the rendering.
I am a bit confused why the method you are using right now would not also show those artifacts since you are still doing floating point computations within the same ranges.
That is because here, it's not the cockpit (which has small details that may move two or three pixels with an inaccurate float) that moves, but the lights that are moved to relative space (and thus are potentially a negligible distance closer or farther away), while the cockpit stays perfectly still at 0,0,0
Converted to draft to investigate if the cockpit pass might now need to happen before show_ship to avoid the deferred render pass to badly overlay over the external ship.
Found the issue: Turns out that after the first deferred lighting pass, all subsequent passes will need all their deferred buffers cleared and not only the base color buffer, as otherwise previously deferred-rendered objects will have effects on the following deferred pass, which is obviously wrong. The buffer can however not always be cleared, as some things (like missile trails), rely on having the buffer set for the first deferred rendering pass.
I looked through the code again and it seems like all lighting computations are done in view space. Since the cockpit is already rendered relative to the eye point I do not see where the floating point inaccuracies could come from.
Uh, could you elaborate on this more? The cockpit is not rendered relative to the eye point as far as I can tell. It's always rendered at 0,0,0, only offset by the cockpit_offset, meaning that all position floats usually stay very low, usually even lower than 10. Whereas when it's rendered in worldspace, numbers can easily approach 10000 or even higher And precision is needed down to around 0.001 in terms of position of the cockpit, as it's close enough to the viewer that a movement by that small amount moves the cockpit by a few pixels. Most other things, like lights or other models, that are done in viewspace, are far enough away (like, a few meters or even more), as so that the effect of this level of precision is lower than one pixel of movement on the screen, hence not being visible if inaccuracies exist. As a matter of fact, I was able to observe the same vibrations for generic object models at 10km distance to the origin. It's not as noticable, since the near plane of the renderer cuts actual objects off in the region where it's actually critical (where the cockpit is rendered at these distances, as the near cutoff plane is much closer in that renderpass). Hence I don't think it's really possible to do within the same coordinate system, and hence, the same render pass
The cockpit is rendered in such a way that the view matrix has no offset so effectively the cockpit is rendered "relative" to the eye point. With that it should even be possible to just render the cockpit "normally" inside the deferred rendering pass since the view space coordinates should be correct for the lighting to work. You might have issues with fog being incorrectly applied to the cockpit but that can be worked around by using the alpha channel in the emissive scene texture to signal if fogging should be applied or not.
Did you look into such a solution? Debugging tools such as RenderDoc could also be useful here to determine if the values that are being written into the various buffers support the assumptions made here.
view matrix has no offset so effectively the cockpit is rendered "relative" to the eye point.
This is exactly the point, why cockpits are different from normal objects.
Normal objects use
gr_set_view_matrix(&Eye_position, &Eye_matrix);
as view matrix. This matrix is relative to the eye pos, as you describe. The cockpit uses
gr_set_view_matrix(&leaning_position, &Eye_matrix);
as view matrix, with leaning_position is mostly just the null vector. This matrix is not relative to the eye pos, thus being a very different view matrix to the general rendering pass.
This second matrix allows the cockpit to be rendered at 0,0,0, independent of what the eye pos is.
That means to display the cockpit in the first matrix, it would need to be moved to the eye pos to be rendered in the same place every frame. As the eye pos get's large with rising distance of the player to the world origin, the needed precision to move the cockpit to the eye pos is not sufficient, and it starts to be a few pixels inaccurate. And since that inaccuracy is not consistently the same for different positions, moving at a distance from the origin gives the impression of the cockpit vibrating. This is confirmed by the fact, that the very same issue is seen for normal objects that are rendered using the first matrix. But due to the fact that two objects in this close proximity rarely ever move exactly the same (as the cockpit would do to the eye pos), it's not noticable in actual gameplay.
From this, we can take away, that we must not render the cockpit (and the ship_render_show_ship_cockpit call btw, which is also rendered relative to 0,0,0 and not to the eye pos) relative to the eye pos.
I think we are using different definitions of "eye point", sorry about that. When I say that I mean the OpenGL definition where the eye is always at 0,0,0 and "view space" positions are relative to that.
That property can be used to render objects "relative" to the eye position by simply setting up the view matrix with a zero vector as the position and then rendering your model relative to the "world position" 0,0,0. Since all lighting operations are done in view space, the actual world position of a vertex does not matter and the lighting should still be correct.
Okay, after lots of tinkering and testing: While you can get deferred lighting to work this way, it will not work in conjunction with shadows. This is because, for the deferred lighting pass, the shadows are applied during the pass, with one set of shadow matrices. However, the shadow view matrix is dependent on the eye_position of the shadow render pass. That means, that the shadows made by the cockpit and these by the rest of the scene cannot be rendered with the same deferred pass without massively reworking how shadows work. And I don't know how they work well enough to do that. That suggestion however did enable me to do the second deferred pass a bit more cleanly, so thanks. And as for concerns that it's more performant with only one deferred pass, and thus would be better: I ran a trace over a really lights heavy mission, and the second lighting pass averages at around 0.14ms (for comparison, this mission takes 20 to 50ms with the profiler on to render the particles alone, and 100 to 140ms for rendering the basic scene. Runs at 60+fps with the profiler off). Which is really not a problem and quite the insignificant amount of time for the frame. So yes, having a second render pass is not optimal. But it is not enough to warrant a change this significant to the shadows, neither in terms of work needed, nor in terms of possible regressions.
I'll also add in there that having deferred lighting work for cockpits makes an extremely substantial difference in immersive gameplay. Also, running stress test missions with fps counter on showed no visible slowdown on my system. Linked videos to show this effects.
https://user-images.githubusercontent.com/14077810/119275868-8482c680-bbe5-11eb-8b4f-677631520804.mp4
https://user-images.githubusercontent.com/14077810/119275871-85b3f380-bbe5-11eb-8b45-bd48bcd80957.mp4
https://user-images.githubusercontent.com/14077810/119275872-86e52080-bbe5-11eb-9441-509c4276511b.mp4
I do not like doing this but, in my opinion, this PR is not mergable in its current state.
Adding the additional deferred lighting pass adds complexity and maintenance overhead that will make future changes to the graphics pipeline more complicated. I have laid out a solution on how to make the cockpits work with the normal deferred pass with relatively little effort.
It should also be possible to render the cockpit in the normal shadow pass since those are also done in view space (since we luckily do not have point light shadows yet) so here you can also circumvent the floating point inaccuracy issue.
If you absolutely want to have shadows in the cockpit then more development effort is needed in that direction. Otherwise you have to choose between shadows and point lights for the cockpit.
Just to comment on this again: Yes, it is theoretically possible to do it the way you suggest, but needs a considerable overhaul of how shadows currently work, plus a lot more masking in different passes to make sure that nebulas and other effects currently rendered in between the objects and the cockpits still behave as they should, despite the order being messed up if the cockpit is rendered during the first render pass. Which arguable adds actually more complexity over just adding a second pass, which only really calls the same deferred render function again + clearing the buffer, so doing the second pass is definitely cleaner codewise than forcing everything into one. Even modifying the deferred pass after this is nor really impeded: This method does not require the deferred pass to work in a certain way. While I see that you want to keep that part as simple as possible, this really does not add that much complexity for a fairly big improvement in resulting gameplay quality and immersion.
And in my humble opinion, the added maintenance overhead is a fraction of the additional complexity we'd occur by having to add a new mask for the cockpit if we'd have to fit it in the main object render pass.
I know this is going to come off as an unpopular opinion, but this has been waiting long enough and with the desire for a few mods to use this feature, I am willing to compromise on merging this in it's current state with a few conditions.
This could be done, with enough finesse, through some render queue finaggling. @qazwsxal and I talked about this at length for a bit and we came to the conclusion that placing the cockpit model at the end of the render queue and making a special case in the shaders to handle cockpits through model and view matrix transforms could achieve the same effect without the overhead and complexity of an additional deferred pass.
@SamuelCho and @asarium might know some of the specifics on the feasibility of doing it this way.
Thus, the condition for merging this as is would be a high priority TODO issue making this other change. Alternatively, you could defer merging of this and take a crack at our suggestion @BMagnu
Unless there are further points of opposition, as discussed, I will be merging this PR in a day or two. Afterwards, I'll put up a high priority Issue for optimizing the render pipeline in the discussed regards.
TLDR of plan from Discord discussion and above:
- Merge deferred cockpit PR,
- Create and merge PR fusing cockpit and extmodel rendering,
- Create and merge PR that combines these two render passes.