GoogleWorkspaceReports Data Connector - Time span bug

[eivhel]

Describe the bug Pull Request #9561 introduced a flaw. The connector will only query for a 15 minute timespan of logs for each run. This means that when first enabling the connector you will only get 15 minutes of log which are 24 hours old. Then it will take many runs to get up to date, since your logs will increment with 5 minutes per run. This is a poor way of reducing load. You should rather specify a lower amount in maxResults and build a logic for how many pages you iterate.

To Reproduce Steps to reproduce the behavior:

1. Deploy the connector in a fresh environment with no prexisting logs.
2. Wait for the conntector to initiate.
3. Query any Workspace log, sorting from newest to oldest.
4. See that the newest log is about 24 hours old. If there was any logs at all in this 15 minute interval.

Expected behavior I expect the conntector to immidiately get up to date on logs and/or start pulling logs from the time the connector is deployed.

Additional context We have deployed this application without line 334-336 and it works ok. I can see the need to limit load if there is a lot of data. But please fix it in a manner which keeps logs up to date.

[github-actions[bot]]

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

[v-muuppugund]

Hi @eivhel , Thanks for flagging this issue, we will investigate this issue and get back to you with some updates by 25-12-2023. Thanks!

[v-sudkharat]

Hi @eivhel, Our team is checking on this issue and also working on repro this. we will share update with you by - 02-01-2024 Thanks!

[v-muuppugund]

Hi @eivhel , Agreed done the initial analysis ,We can remove it as we are introducing the 24 hours delay on initial run and reduce the max results ,will make max results as configurable,will be working on the changes,will update you,will add an additional logic for not initial run and data is older than some time,so we are not fetching the enitre data.

[eivhel]

@v-muuppugund That sounds great. Thank you.

[v-muuppugund]

Hi @eivhel ,Done the changes locally ,working on testing it, Will need some more time,will update you once the testing completed

[v-muuppugund]

Hi @eivhel ,need some more time for completing the testing and will update you once PR is pushed

[v-muuppugund]

Hi @eivhel ,Local testing completed,working on deployment and testing,once done ,will share the package for testing

[v-muuppugund]

Hi @eivhel ,Sill need some time for completing testig ,will update you once done

[v-muuppugund]

Hi @eivhel ,Working on the revalidating the issue as there are changes pushed in master ,will get back to with an update by 21Mar24

[v-muuppugund]

Hi @eivhel ,This issue has been fixed and addressed in V2 version of it ,so closing the issue, If you still need support for this issue(https://github.com/Azure/Azure-Sentinel/issues/9637), feel free to re-open at any time. Thank you for your co-operation!