[feature]: Add support for Application across test sets

[slayerjain]

Is there an existing feature request for this?

• [X] I have searched the existing issues

Summary

We need to add support for restarting the application only if there is a config change.

Why should this be worked on?

1. Restarting the application for every test set is over kill (because it can become slow for slow starting apps).
2. Application test sets may require different config.

Repository

keploy

[re-Tick]

I think there can be config calls or the initialisation external calls (like migration or settings calls) which can change overtime. And if the behaviour of the API depends on the state of the config/initialisation calls, then the app needs to restarted. So, we need to compare the initialisation calls of the other test-sets and restart the application if the config/initialisation calls are unmatched.

[charankamarapu]

There 3 type of changes that can be done in an application

1. code change
2. property changes(like change in application.properties file/.env file etc)
3. both

Majorly in case of code change user must re-record the the test cases there is no alternative because the DB calls will change. So if two test-sets are recorded in different point of time of the application the first recorded test-set will fail even after re-running the application because the code itself is changed.

Coming to property changes if two test-sets are recorded with property changes like one is recorded with local-env-properties and other with dev-env-property. Then also even if we re-run the application for each test-case the keploy tests will fail unless the user provides that property file to the test-set via app command or env variable at that exact moment the first/second test-set is about to be started.

By considering these situations even if we remove the rerunning of the application now there will no effect on the behaviour of the keploy.

considering the feature request to re-run the application in case of config changes, generally property changes will/should be concatenated to the application command like in java while running a jar we will specify the profile which you want to run with the jar. To rerun the application while there is a property change the command will also be different as property file will be attached to it. So associating the application cmd with test-sets field in the keploy-config file will actually resolve this.

Please provide your thoughts on this.

[slayerjain]

I agree. but there is a trade off to consider here.

1. if we define the mapping of test-sets to app config/cmd in the centralised Keploy-config. It would result in unnecessary merge conflicts when different devs try to edit the same Keploy-config wit their own tests.
2. While if we make this config part of the test-sets, It becomes harder from user onboarding point of view. This config may not be that obvious unless we create a guided process to manage this.

[re-Tick]

I also agree @charankamarapu. Re-running of the application is not required.

[charankamarapu]

Got it @slayerjain . I will be working creating a config.yaml file for every test-set. If the cmd while testing is different from the global config we will create a cmd param in the config.yaml inside the test-set and use it instead of the global command for testing.