Add feature to capture on miss in spy mode

[kapishmalik]

Resolves #864

@tommysitu I have added a feature to capture on miss in spy mode. Could you please review. If it looks fine, then I will add documentation and few tests in same PR.

[kapishmalik]

@tommysitu can you check this one. If it looks fine then I will add UTs and documentation around the same.

[ayyasaran]

Is this feature available on the latest hoverfly version? When I tried, I am getting below message.

unknown flag: --capture-on-miss

[kapishmalik]

No it is still in PR.

[ayyasaran]

No it is still in PR.

Thanks for your response. Is this expected to be available soon? Any ETA?

[johenning]

What is the current blocker of this PR? Any way of contributing to a resolution?

[tommysitu]

@johenning not sure how useful this feature is, but I can review it again.

[johenning]

@johenning not sure how useful this feature is, but I can review it again.

Just in case this was meant as a question: Yes that would be great.

I need this feature for my usecase, I can go into detail if that helps.

[tommysitu]

@johenning yes pls, could you let us know your use case? Will see if having a flag on the spy mode is a sensible way to implement this feature.

[johenning]

We use Hoverfly for our E2E tests. We have multiple microservices that talk to each other directly and talk to systems outside of our control through Hoverfly. In PRs Hoverfly used to run in replay mode so that any missing recording would fail the tests. On main we have Hoverfly run in spy mode, because even when the tests are running, we want to allow parallel use of our dev landscape. We have headers to differentiate test-tenants from others, so only the test tenants gets their requests replayed. This all worked well until recently when we introduces another UI-based E2E testssuite. It has a different scope and framework than the API-based E2E tests, so it makes sense to have them separate. It also does not use Hoverfly, the underlying systems used in these tests are stable enough (these systems can't be differentiated solely via their URLs). But they talk to the same underlying microservices and both testsuites take rather long to run, so we want to be able to run them at the same time. Now on main that still works, but not in PRs. In PRs I want to ensure that:

• E2E and UI-E2E tests can run at the same time
• UI-E2E tests are unaffected by Hoverfly
• Any missing request in the E2E tests fails the test suite
• while both testsuites are indirectly running against the same Hoverfly instance

My envisioned solution (which I already build, misunderstanding what spy mode does) is that I run Hoverfly in spy mode with E2E-test recordings loaded and capture any new request. After the tests have run I download the recordings, discard the UI-E2E test related ones (which I can differentiate based on a header) and diff the remainder against the stored recordings. If there are any changes, I fail the testsuite and require re-recording for the affected tests.

So far, allowing capture in spy-mode seems like the best solution to me. I'm open to other ideas if you have some. Not sure if a flag is the right solution, or if this should be a its own SpyCapture mode. For our usecase it doesn't really make a difference. I would just have the PR deployment have the flag and leave main as is.

[tommysitu]

Thanks for the details! I'm wondering if you have tried the diff mode, it's built exactly for your use case in mind. The diff mode is supposed to monitor your simulation against the real environment, if discrepency is detected, you can then instruct Hoverfly to re-record your simulation.

[tommysitu]

@kapishmalik my hold back on this implementation is because spy mode isn't intended for making change to simulation, it's a term borrowed from Mockito which means "real partial mock". It's a hybrid mode allowing user to mock just some of the APIs while serving traffic for the rest. It could become confusing if we start overloading this mode with more meanings or purposes.

So currently if user wants to make a change to a simulation, they should explicitly use capture mode, it's simple and straight forward by recording anything that is not yet in the simulation. It's not worth to complicate this behaviour.

So if people want to capture unmatched requests rather than failing the test during a simulation, it sounds like they just want to use Hoverfly as a cache? It would make more sense to create a new mode and call it cache mode, or something cooler like autopilot mode as it seamlessly switch between simulate or capture without human intervention.

[johenning]

Thanks for the details! I'm wondering if you have tried the diff mode, it's built exactly for your use case in mind. The diff mode is supposed to monitor your simulation against the real environment, if discrepency is detected, you can then instruct Hoverfly to re-record your simulation.

Yes, we use the diff mode for drift detection. Problem is, correct me if I'm wrong, that requests are not replayed in diff mode. We have several tests for the API that won't work with a live system, because they test e.g. maintenance error messages, timeouts, or manually constructed data, i.e. these tests only work when replaying and can't be recorded (or only during short time periods). Thats why I only want to record requests that are not already in the recordings

[tommysitu]

Diff doesn't do a replay, and it calls through your live system. However, with this PR, it's gonna record everything to your live system and modify your simulation, is it something you want?

[johenning]

Diff doesn't do a replay, and it calls through your live system. However, with this PR, it's gonna record everything to your live system and modify your simulation, is it something you want?

No, thats not what I want. I don't really know enough about the codebase to understand what this PR is doing. My assumption was that the CaptureOnMiss mode only captures when there is not match in the recordings.

[tommysitu]

To illustrate this feature, if you have a simulation containing this initially:

Request A
Request B

When you run your test with the -capture-on-miss flag in spy mode, and you make these requests:

Request A (simulated) 
Request B (simulated)
Request C (not matched - proxied through and captured)
....
Request X (not matched - proxied through and captured)

At the end of all your test run, you will end up with a simulation containing everything proxied by hoverfly in your tests:

Request A  
Request B 
Request C 
....
Request X 

This is what it will do.

[johenning]

However, with this PR, it's gonna record everything to your live system and modify your simulation

Ok, then it seems I misunderstood this line. It doesn't record everything then, just non-matched requests.

This is what it will do.

That looks exactly right. I would then take the new recordings:

Request A  
Request B 
Request C 
....
Request X

filter our the ones I don't care about:

Request A  
Request B 
Request C 

and diff it against my stored recordings, i.e. the inital simulation:

Request A
Request B

If they match - perfect. If not I fail the tests and log the diff

+Request C 

This would be exactly what I'm looking for and AFAIK I can't currently accomplish this with any of the existing modes

[tommysitu]

Looks like the branch has been deleted, waiting for @kapishmalik to raise a new PR again.

[kapishmalik]

@tommysitu added new PR. https://github.com/SpectoLabs/hoverfly/pull/1126

[kapishmalik]

@johenning feature is on master branch.