Notify via github comment if tests passed in retry

[Talank]

We already implemented this mechanism in some other repos such as in web. For example as shown in the image below.

We should first detect if the GUI tests passed in the first attempt or in the retry. If it passes in retry then it is identified as flaky. We should create an issue to fix the detected flaky tests.

[Talank]

In desktop client, we generate gui test report in json format. Currently, we are using parser to clean up the report and only show the details of failed tests. However, if a test failed in retry, then it actually has 2 entries (assuming that retry is done only once) i.e one for failed and one for passed test. We somehow missed it while designing the parser. However, I ran a test (forcefully making it flaky) and noticed 2 entries. If we find 2 entries for a same test (i.e combination of test file, scenario and step), with different result, then we can be sure that the test passed in retry. For this purpose, we need to update our gui test result parser. For test data during the update, we can test on this log, where it should say that the scenario logging out passed in retry.

example test data where one test failed in retry: results.zip

Here is the detail of test passing in retry in the above test data:

  "feature": "Logout users",
  "scenario": "logging out",
  "step": "When the user 'Alice' logs out of the client-UI"
  "error": "Comparison	'1' and '11' are not equal"

If we run or current parser in the above test data, then it shows us that 1 test (the above test) has failed. Nevertheless, the test fail is from the first attempt. And there is test pass in the second attempt.

[Talank]

For the sake of getting nice test result in html and js files, we had to disable json test reports. Thus, we either need to redesign the testparser or we need to wait for the feature to generate testresult both in html and json is implemented in squish by froglogic. And untill one of those thing is executed, this issue is blocked.

REF: https://github.com/owncloud/client/issues/9454

Note if we want to redesign the parser by reusing the data generated while creating the html,js report, then it might actually be easier to get the retried tests. I have extracted the result of html file in this json file: GUIreport.zip

[Talank]

Irrespective to the json/html report data, the test context itself contains test names, and status of test run. We should be able to make use of these data and determine if a certain test has passed in retry or not. Most recently, these data has been used to take screenshot of desktop GUI in case of test errors.

[saw-jan]

Whoever takes this issue in the future, most of the hard work has been done already. See https://github.com/owncloud/client/pull/9584

[saw-jan]

github token won't be exposed to the PRs due to security reasons. So this won't be possible to implement