Fluids - Compute component-wise error for NS solver

[laylagi]

This is part of a course project in collaboration with @jnclement and @amandashack . It adds support for computing the error for each component and converts the errors to the other set of variables (primitive <-> conservative). It currently breaks for the other solvers though (advection and euler).

[laylagi]

I think this branch was working for our purposes in the course project but is broken for the Advection2D and Euler solvers. A quick fix would be to define a convert_state QFunction for each of those solvers. We could, instead, refactor the Euler problems to use newtonian.c (could we do the same thing with Advection?). However, the branch would diverge from its original purpose.

[jedbrown]

Hmm, I think State{,Primitive,Conservative} need not be "Newtonian" things (they have nothing to do with viscosity). Their conversions assume ideal gasses. So we could promote that concept to something that Euler would also use. That's scope creep and I think it would also be fine to just disable your feature for Advection and Euler.

[laylagi]

Oh, true! I was talking about Newtonian since that is the only solver that uses NewtonianIdealGasContext, IIRC. Then, I would prefer disabling the feature for Advection and Euler for now as we have decided to refactor Euler (to use Newtonian) anyway.

[jedbrown]

Sounds good to me. Newtonian is what matters for us (and at some point in the next few months, I think we'll add RANS).

[laylagi]

@jedbrown , does this look good to you now?

[laylagi]

@jedbrown , can we merge this branch if there are no other concerns?

[laylagi]

Here is the output with the traveling vortex problem:

$ build/fluids-navierstokes -problem euler_vortex -degree 2 -q_extra 2 -dm_plex_box_faces 5,5,1 -dm_plex_box_lower 0,0,0 -dm_plex_box_upper 125,125,250 -dm_plex_dim 3 -mean_velocity 1.4,-2.,0 -bc_inflow 4,6 -bc_outflow 3,5 -bc_slip_z 1,2 -ts_dt 1e-7 -ts_rk_type 5bs -ts_rtol 1e-10 -ts_atol 1e-10 -ts_max_time 1

-- Navier-Stokes solver - libCEED + PETSc --
  MPI:
    Host Name                          : leila-ThinkPad-P53s
    Total ranks                        : 1
  Problem:
    Problem Name                       : euler_vortex
    Test Case                          : isentropic_vortex
    Background Velocity                : 1.400000,-2.000000,0.000000
    Stabilization                      : none
  libCEED:
    libCEED Backend                    : /cpu/self/opt/blocked
    libCEED Backend MemType            : host
  PETSc:
    Box Faces                          : 5,5,1
    DM MatType                         : aij
    DM VecType                         : standard
    Time Stepping Scheme               : explicit
  Mesh:
    Number of 1D Basis Nodes (P)       : 3
    Number of 1D Quadrature Points (Q) : 5
    Global DoFs                        : 1573
    Owned DoFs                         : 1573
    DoFs per node                      : 5
    Global nodes (DoFs / 5)            : 314
    Local nodes                        : 363
Time taken for solution (sec): 1.41848

Relative Error (absolute error norm, true solution norm):
  Conservative variables:
        Component 0: 0.0471095 (0.047109, 0.99999)
        Component 1: 0.364438 (0.505128, 1.38605)
        Component 2: 0.133585 (0.299979, 2.24561)
        Component 3: inf. (5.52411e-16, 0.)
        Component 4: 0.0467154 (0.264258, 5.65677)

Time integrator took 245 time steps to reach final time 1.00194

[jedbrown]

Why does it always say Conservative variables (regardless of -state_var primitive) and wouldn't we expect this to converge under refinement? Also, the domain shape (dm_plex_box_upper 125,125,250) relative to resolution dm_plex_box_faces 5,5,1 looks weird.

[laylagi]

Why does it always say Conservative variables (regardless of -state_var primitive) ?

It's because traveling vortex doesn't support primitive variables yet.

[laylagi]

wouldn't we expect this to converge under refinement?

I remember we didn't see convergence with -degree 2 in the summer. I only presented degree 1 in my talk.

[laylagi]

Also, the domain shape (dm_plex_box_upper 125,125,250) relative to resolution dm_plex_box_faces 5,5,1 looks weird.

I took the arguments from the regression tests in navierstokes.c (and refined it) to show you the output. Oops, it is indeed weird!

[jedbrown]

I wonder what happened. I thought we had made a convergence test (long ago) for this problem at arbitrary order. Anyway, how does isentropic vortex test your to_prim conversion if it's using exclusively conservative variables? Are the printed norms before or after conversion?

And with -options_left, I see this (which is good).

Option left: name:-state_var value: primitive source: command line

[laylagi]

how does isentropic vortex test your to_prim conversion if it's using exclusively conservative variables? Are the printed norms before or after conversion?

I think we decided to report the component-wise norms for isentropic vortex as well. This problem doesn't have to_prim conversion and it's printing the norms without any conversion.

[jedbrown]

Sounds good. How do we test the new feature?

[laylagi]

Hmm, the conversion doesn't seem to be right! They did make sense to me when I was running experiments for the course project (although I wasn't running them with the same arguments)!

$ build/fluids-navierstokes -options_file examples/fluids/tests-output/blasius_stgtest.yaml -state_var primitive -ts_dt 2e-7 
...
Time taken for solution (sec): 0.312749

Relative Error (absolute error norm, true solution norm):
  Primitive variables:
        Component 0: 0.12287 (12409.8, 101000.)
        Component 1: 5.03298 (201.319, 40.)
        Component 2: 73.5649 (11.072, 0.150506)
        Component 3: -nan. (0., 0.)
        Component 4: 0.111034 (31.9779, 288.)

Time integrator took 5 time steps to reach final time 1e-06

$ build/fluids-navierstokes -options_file examples/fluids/tests-output/blasius_stgtest.yaml 

...
Time taken for solution (sec): 0.233436

Relative Error (absolute error norm, true solution norm):
  Conservative variables:
        Component 0: 0.000671353 (0.000820348, 1.22193)
        Component 1: 0.0109931 (0.537314, 48.8773)
        Component 2: 2.20481 (0.405483, 0.183908)
        Component 3: -nan. (0., 0.)
        Component 4: 0.000997987 (252.792, 253302.)
  Primitive variables:
        Component 0: 1.00067 (404276., 404006.)
        Component 1: 93.064 (14890.2, 160.)
        Component 2: 6584.88 (3964.26, 0.602025)
        Component 3: -nan. (0., 0.)
        Component 4: 68.2968 (78679., 1152.02)

Time integrator took 5 time steps to reach final time 1e-05

[laylagi]

This is rebased now but when I try to reproduce these runs, I see a segmentation violation. Same issue exists on main as well. I'm investigating. Also, what sort of test are you thinking of? Something that could reproduce the conversion?

[jedbrown]

Probably the issue just fixed by https://github.com/CEED/libCEED/pull/1179.

[laylagi]

Probably the issue just fixed by #1179.

Yep! That issue is fixed but all the blasius tests and "turb_spanstats" are failing now. I haven't been following the developments in the fluids mini-app recently. The tests are not broken (which make the debugging more difficult), they are just failing with some norm. Can you think of any recent changes related to the blasius problem that could cause the failure?

[jedbrown]

Failing in this branch or on main? @jrwrigh has been dealing with a CI issue.

[jrwrigh]

My issues with the CI are limited to the ROCM and float jobs, not to the fluids examples specifically.

As for the failures, the fluids problems aren't failing on main, but I also don't see a reason why any changes made in this PR should affect the testing; The binary files and RegressionTests_NS are untouched, and it doesn't look like the solution data is being messed with.

[laylagi]

The tests wouldn't fail until I rebased the branch onto main. It is so confusing 🤷🏼‍♀️ . I will try to look into it more closely today.

[laylagi]

Alright, I think this approach is working now! I am trying to perform a convergence study but a quick test shows that at least the conversion is correct:

$ mpiexec -n 6 build/fluids-navierstokes -options_file examples/fluids/tests-output/blasius_test.yaml -ts_dt 1e-9 -ts_max_steps 20 -state_var conservative

...

L2 Error:
  Conservative variables-Component 0: 3.86413
  Conservative variables-Component 1: 2.57201
  Conservative variables-Component 2: 1.66507
  Conservative variables-Component 3: 2.55199
  Conservative variables-Component 4: 3.7305
  Primitive variables-Component 0: 0.98822
  Primitive variables-Component 1: 1.49868
  Primitive variables-Component 2: 1.54276
  Primitive variables-Component 3: 1.06039
  Primitive variables-Component 4: 0.66525
Time integrator CONVERGED_ITS on time step 20 with final time 2e-08

$ mpiexec -n 6 build/fluids-navierstokes -options_file examples/fluids/tests-output/blasius_test.yaml -ts_dt 1e-9 -ts_max_steps 20 -state_var primitive
...

L2 Error:
  Primitive variables-Component 0: 0.988239
  Primitive variables-Component 1: 1.49869
  Primitive variables-Component 2: 1.54278
  Primitive variables-Component 3: 1.06041
  Primitive variables-Component 4: 0.665282
  Conservative variables-Component 0: 3.86417
  Conservative variables-Component 1: 2.57205
  Conservative variables-Component 2: 1.66509
  Conservative variables-Component 3: 2.552
  Conservative variables-Component 4: 3.73052
Time integrator CONVERGED_ITS on time step 20 with final time 2e-08

(Just squashed the commits to make the review easier.)

[laylagi]

Thanks, I added comments for q_true and changed the names. I am happy with a squash-merge if this last bit looks good to you.

[laylagi]

Can we have one test of this functionality?

Do you mean a convergence plot?

[jedbrown]

Well, something that runs this code and checks that the errors are of expected size. (What if a bug caused the reported error to be zero always or the error became huge?) Ideally we would make a convergence plot for the isentropic vortex or channel flow, but I can handle that not being ready.

[jedbrown]

Cool. Squash or squash-merge?

[laylagi]

Thanks, @jrwrigh for bearing with me. A lot has changed since the last time I contributed to this mini-appand. There are still a couple of remaining tasks before merging:

• [x] Adding a test that Jed mentioned above
• [x] Fixing the broken code for traveling vortex and 2d advection
• [ ] Fixing the bug in reporting the error (the computed error depends on the mpi rank)

[laylagi]

Cool. Squash or squash-merge?

squash-merge is fine after I am done with the remaining tasks.

[laylagi]

I am seeing a weird issue. The errors are not correct with rank 1.

$ build/fluids-navierstokes -problem euler_vortex -degree 3 -dm_plex_box_faces 1,1,2 -dm_plex_box_lower 0,0,0 -dm_plex
_box_upper 125,125,250 -dm_plex_dim 3 -units_meter 1e-4 -units_second 1e-4 -mean_velocity 1.4,-2.,0 -bc_inflow 4,6 -bc_outflow 3,5 -bc_slip_z 1,2 -vortex_strength 2 -ksp_atol 1e-4 -ksp_rtol 
1e-3 -ksp_type bcgs -snes_atol 1e-3 -snes_lag_jacobian 100 -snes_lag_jacobian_persists -snes_mf_operator -ts_dt 1e-3 -implicit -dm_mat_preallocate_skip 0 -ts_type alpha


L2 Error:
  Conservative variables-Component 0: 0.000741784
  Conservative variables-Component 1: 0.000961908
  Conservative variables-Component 2: 0.00153698
  Conservative variables-Component 3: 0.
  Conservative variables-Component 4: 0.00399707
Time integrator CONVERGED_TIME on time step 50 with final time 0.05

$ mpiexec -n 4  build/fluids-navierstokes -problem euler_vortex -degree 3 -dm_plex_box_faces 1,1,2 -dm_plex_box_lo
wer 0,0,0 -dm_plex_box_upper 125,125,250 -dm_plex_dim 3 -units_meter 1e-4 -units_second 1e-4 -mean_velocity 1.4,-2.,0 -bc_inflow 4,6 -bc_outflow 3,5 -bc_slip_z 1,2 -vortex_strength 2 -ksp_at
ol 1e-4 -ksp_rtol 1e-3 -ksp_type bcgs -snes_atol 1e-3 -snes_lag_jacobian 100 -snes_lag_jacobian_persists -snes_mf_operator -ts_dt 1e-3 -implicit -dm_mat_preallocate_skip 0 -ts_type alpha

L2 Error:
  Conservative variables-Component 0: 2.66349e-06
  Conservative variables-Component 1: 8.8622e-05
  Conservative variables-Component 2: 6.99578e-05
  Conservative variables-Component 3: 3.72699e-155
  Conservative variables-Component 4: 2.05213e-05
Time integrator CONVERGED_TIME on time step 50 with final time 0.05

$ mpiexec -n 8  build/fluids-navierstokes -problem euler_vortex -degree 3 -dm_plex_box_faces 1,1,2 -dm_plex_box_lower 0,0,0 -dm_plex_box_upper 125,125,250 -dm_plex_dim 3 -units_meter 1e-4 -units_second 1e-4 -mean_velocity 1.4,-2.,0 -bc_inflow 4,6 -bc_outflow 3,5 -bc_slip_z 1,2 -vortex_strength 2 -ksp_atol 1e-4 -ksp_rtol 1e-3 -ksp_type bcgs -snes_atol 1e-3 -snes_lag_jacobian 100 -snes_lag_jacobian_persists -snes_mf_operator -ts_dt 1e-3 -implicit -dm_mat_preallocate_skip 0 -ts_type alpha

L2 Error:
  Conservative variables-Component 0: 2.66349e-06
  Conservative variables-Component 1: 8.8622e-05
  Conservative variables-Component 2: 6.99578e-05
  Conservative variables-Component 3: 4.56152e-155
  Conservative variables-Component 4: 2.05213e-05
Time integrator CONVERGED_TIME on time step 50 with final time 0.05

(same issue with advection2d and blasius) It could be the logic in FixMultiplicity_NS() but I can't point at anything specific.

[laylagi]

Testing with different arguments/problems, I see that the results could be different for n>1 as well.