Some test failures are not written

[Starkmann]

I'm using junit to collect tests result and not all tests results are written.

It seems like only the last process (suite) is writing the results.

My behat.yml:

default:
  formatters:
    junit:
      output_path: "tests/behavior/build"
    pretty: true
  suites:
      default:
        filters:
          tags: "~@skip"
        #@TODO: split motion and newsletter
        paths:
          features: '%paths.base%/features/Motion'
        contexts:
          - DrevOps\BehatScreenshotExtension\Context\ScreenshotContext
          - FeatureContext
      incident:
        filters:
          tags: "~@skip"
        paths:
          features: '%paths.base%/features/Incident'
        contexts:
          - DrevOps\BehatScreenshotExtension\Context\ScreenshotContext
          - IncidentContext
      proposal:
        filters:
          tags: "~@skip"
        paths:
          features: '%paths.base%/features/Proposal'
        contexts:
          - DrevOps\BehatScreenshotExtension\Context\ScreenshotContext
          - IncidentContext
      seminars:
        filters:
          tags: "~@skip"
        paths:
          features: '%paths.base%/features/Seminars'
        contexts:
          - DrevOps\BehatScreenshotExtension\Context\ScreenshotContext
          - SeminarsContext
      myMotions:
        filters:
          tags: "~@skip"
        paths:
          features: '%paths.base%/features/MyMotions'
        contexts:
          - DrevOps\BehatScreenshotExtension\Context\ScreenshotContext
          - MyMotionsContext
  extensions:
     DrevOps\BehatScreenshotExtension:
        dir: tests/behavior/screenshots
        fail: true
        purge: true
     Behat\MinkExtension:
        base_url: https://xxx.lan
        browser_name: chrome
        selenium2:
          wd_host: http://browser:4444/wd/hub
          capabilities: {
            "browser": "chrome",
            "version": "*" ,
            "acceptSslCerts": true,
            "extra_capabilities": {
              "acceptInsecureCerts": true,
              'chrome': {
                args: [
                  '--disable-dev-shm-usage',
                  '--window-size=1920,6080',
                ],
              }
            }
          }
        files_path: tests/behavior/dummy/
     DMarynicz\BehatParallelExtension\Extension: ~
jenkins:
  formatters:
    junit:
      output_path: "tests/behavior/build"
    pretty: true

In this case i have errors in every suite, which is written to each xml file when i run without parallel. In parallel only errors of suite myMotions is written to xml.

[Daniel-Marynicz]

Because parallel executes behat multiple times at once. And if it is to be saved to the same xml in each run, then it gets overwritten many times .

[Starkmann]

OK, im using --parallel-feature like this: vendor/bin/behat --parallel-feature 4 --colors -c tests/behavior/behat.yml So i would need to write one xml per feature, right?

[Daniel-Marynicz]

Yes this should work. If junit this supports.

[Starkmann]

This is not supported by behat at all. With vendor/bin/behat --parallel-feature -f junit -o xml -f pretty -o std -c tests/behavior/behat.yml i at least get every suite written, but in each suite only the last failure, as you said.

[Daniel-Marynicz]

It is likely that only the junit formatter has this problem. If you try to extend or decorate this formatter, it might work.