allure step doesn't work if is used in global before :all hook

[emaks]

I use allure-rspec 2.17.0 rspec 3.11.0

STR

• configure allure and rspec

AllureRspec.configure do |config|
  config.results_directory = "report/allure-results"
  config.clean_results_directory = true
  config.logging_level = Logger::WARN

  logger = Logger.new($stdout)
  logger.sev_threshold = Logger::WARN
  config.logger = logger
end

RSpec.configure do |config|
  # Enable flags like --only-failures and --next-failure
  config.example_status_persistence_file_path = ".rspec_status"

  # Disable RSpec exposing methods globally on `Module` and `main`
  config.disable_monkey_patching!

  config.expect_with :rspec do |c|
    c.syntax = :expect
  end
end

• create test method

class TestUtils
  extend AllureStepAnnotation

  step("do something")
  def do_something
    # do something
  end
end

• Use this method in before

RSpec.describe Tests do
  before :all do
    TestUtils.new.do_something
  end

  describe "describe_1" do
    it "test_1" do
      # test something
    end
  end
end

Result:

There is no "do something" step in Allure report

Also see errors in console

E, [2022-08-25T16:37:56.794116 #8687] ERROR -- : Could not start test step, no test case is running
E, [2022-08-25T16:37:56.794143 #8687] ERROR -- : Could not start test step, no test case is running

[andrcuns]

This is by design. before :all hook runs outside of test context, meaning test case has not started so there is nothing to attach step to.

[emaks]

@andrcuns for other languages like JAVA, JS allure provides this possibility. Could you suggest workaround for this case?

[andrcuns]

I'm not sure how other frameworks could support it because the problem is in the core concept of how report works.

before :all is executed once and before all of the tests, because allure report doesn't have a concept of a global before hook, there is nowhere to put a step that is executed once before any test case started execution. The only workaround I can imagine is to not use steps in before :all or after :all hooks.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[kudrykv]

@andrcuns containers have befores and afters fields, which in allure UI map to setup and teardown. Yes, the test case did not start yet; however, we still can record and attach those steps to test containers

[andrcuns]

@andrcuns containers have befores and afters fields, which in allure UI map to setup and teardown. Yes, the test case did not start yet; however, we still can record and attach those steps to test containers

Oh, You are correct, for some reason I was completely sure that befores and afters are only present in test result, not container. I will reopen this issue

[andrcuns]

I had to refresh my memory on rspec implementation and while this because allure report doesn't have a concept of a global before hook part is incorrect (since we can add before/after to test container though I'm not sure how it would be displayed in report, haven't tried it), the problem is with rspec itself.

It's very tricky to detect that actual hook is running because rspec doesn't really mark it in the existing events and specific event also doesn't exist for that.

[kudrykv]

I'm playing around a bit with it; maybe, this could be a path forward?

Such methods for lifecycle:

def start_before_test_case(test_result)
  unless current_test_result_container
    return logger.error { 'Could not start before test case, no container is running.' }
  end

  clear_step_context
  logger.debug { "Starting before test case: #{test_result.name}" }

  test_result.start = Allure::ResultUtils.timestamp
  test_result.stage = Allure::Stage::RUNNING

  current_test_result_container.befores.push(test_result)
  current_test_result_container.children.push(test_result.uuid)
  @current_test_case = test_result
end

def update_before_test_case
  unless @current_test_case
    return logger.error { 'Could not update before test case, no before test case is running' }
  end

  yield(@current_test_case)
end

def stop_before_test_case
  unless @current_test_case
    return logger.error { 'Could not update before test case, no before test case is running' }
  end

  logger.debug { "Stopping before test case: #{@current_test_case.name}" }
  @current_test_case.stop = Allure::ResultUtils.timestamp
  @current_test_case.stage = Allure::Stage::FINISHED

  clear_current_test_case
  clear_step_context
  # not saving @current_test_case is intended
end

Those still need updates b/c of the multiple containers, and from what I see, I need to bring a copy of all befores and afters to the last container from previous ones.

And such to the formatter:

def example_group_started(example_group_notification)
  description = example_group_notification.group.description.then { |desc| desc.empty? ? 'Anonymous' : desc }

  lifecycle.stop_before_test_case

  lifecycle.start_test_container(Allure::TestResultContainer.new(name: description))
  lifecycle.start_before_test_case(Allure::TestResult.new(name: description))
end

def example_finished(example_notification)
  lifecycle.update_test_case(&update_test_proc(example_notification))
  lifecycle.stop_test_case
end

[andrcuns]

There might be some complications, because before and after hooks in allure are similar to steps and test cases, it needs to be started and stopped as well so that all the steps in it are detected properly.

Though I'm not sure if it's the same for the ones in test container, a json schema would be nice.

[kudrykv]

1. After example_group_started executes, before-steps start to run.
2. When example_started, all before-steps are executed. After example_started completes, after-steps start to run.
3. When example_group_finished runs, that means after-steps for the given scope run.

In this flow, the biggest pain in the ass is the after-steps. There's no easy way within the current codebase to differentiate between after-steps in different contexts and put them in appropriate containers.

[andrcuns]

If You want to take a stab at this, feel free to propose a PR. I'm afraid I won't have time to try and dig deeper in how to better solve this issue.