oc_config_validate: Support set-replace operations

[xavier-contreras]

Currently tests are hardcoded to use set-update logic, eg:

• https://github.com/google/gnxi/blob/master/oc_config_validate/oc_config_validate/testcases/setget.py#L31C30-L31C43
• https://github.com/google/gnxi/blob/master/oc_config_validate/oc_config_validate/testcases/set.py#L24

It would be great if they could support set-replace as well.

This would particularly be useful / important for the init config -- perhaps that can be the first gap to address:

• https://github.com/google/gnxi/blob/master/oc_config_validate/oc_config_validate/runner.py#L228

Vendors do not always implement these functions well so they are good to test, or certain bugs are triggered based on which method is used.

[JoseIgnacioTamayo]

Ack, somehow we should also offer the possibility to validate configuration changes using SetReplace.

A little background: Set and SetGet tests have the intention to validate that small and puntual config changes happen successfully, with minimal impact to other configuration elements. SetUpdate allows to apply a configuration change without affecting other elements not mentioned in the config blob. Also, SetUpdate is used for the InitialConfig so that only the elements in the provided config are applied, leaving all other elements as they are. Using SetUpdate allows to only change what `oc_config_validate' is told to change.

If we use SetReplace for InitialConfigs, users then would need to provide a larger JSON config, possibly containing elements not related to the tests (If by accident a leaf of container is omitted, it will either be reset to default or deleted).

An approach is to create specifig Testcase Classes SetReplaceGet*, similar to SetGet* that use the SetReplace method, and for those testcases it is clear what behaviour to expect from the Target (keeping the SetGet Testcases with SetUpdate)

The objective of oc_config_validate is to validate that configuration changes with gNMI are correct. oc_config_validate does not intend to validate that gNMI implementations comply with the specification. In that sense, as there are configuration workflows that would use SetReplace specifically, I propose we add specifig Testcases and keep the current behavior for InitialConfigs (to avoid needing larger JSON blobs, or accidentally changing elements not related to the tests)

[xavier-contreras]

Agreed with creating Testcase Classes like SetReplaceGet and leaving current behavior for others as is.

I still think an option to set replace the init config makes sense. Consider the following sequence of events when init is sent as update, assuming a continuous testing scenario. These tests could be telemetry, expecting certain update counts, leafs, etc.

testgroup.yaml - run at 12PM

1. Init config with interface 1
2. Test that validates interfaces, expecting 1
3. Test that adds interface 2
4. Test that validates interfaces, expecting 2

testgroup.yaml - run at 1PM

1. Init config with interface 1
2. Test that validates interfaces, expecting 1 <--this now fails
3. Test that adds interface 2
4. Test that validates interfaces, expecting 2

[JoseIgnacioTamayo]

Ack, I see the need for controlling how the intiial config is applied.

ActionItems:

• Option to control the initial config set method: --init-set-method, defaults to update
• Testcases that specifically use the SetReplace method

[JoseIgnacioTamayo]

I wrote the 2nd part of this Issue: https://github.com/google/gnxi/pull/386

[JoseIgnacioTamayo]

Closing the feature request as completed.

@samribeiro or @mike-albano , could you please close this issue?