Bluepill's list of tests is not the same as the list of tests that get executed

[ravimandala]

We recently found that the list of tests that Bluepill comes up with based on xctest files is not accurate. Bluepill uses nm utility to read symbols from binary from xyz.xctest and picks everything which has a test prefix like this... https://github.com/linkedin/bluepill/blob/42a4b8c9b8d07f58d55872aa45360730a858a768/bp/src/BPXCTestFile.m#L17-L18)

Turns out there is a possibility of a lot of false-positives and true-negatives with this approach. Names of properties in test classes with a test prefix will be wrongly picked up as test cases, and more importantly, if a test class inherits a bunch of test cases from another test class they cannot be identified with this approach.

We are in the process of replacing this with a better and reliable approach that can accurately return a list of test methods that will eventually get executed. (It's worth noting that all the tests that need to be executed will get picked up by the simulators regardless of this issue.)

\cc @ob

[ob]

Thanks for picking this up! Looking forward to the PR :-)

[ravimandala]

I'm working on this. I'll have a PR out in a week.

[ravimandala]

The long-term fix for this is to bring up the simulator and do some code injection magic to get an accurate list of tests. But before that, I implemented a temporary workaround that would factor in test class inheritance and corresponding inherited test cases into account. The PR https://github.com/linkedin/bluepill/pull/368 introduces test time estimate based splitting and incorporates this workaround for both test count based splitting and test time estimate based splitting. Please take a look. \cc @ob

[ravimandala]

Open https://github.com/linkedin/bluepill/pull/376 off v4.1.1+packing branch for this. I will create another PR off the master and then change https://github.com/linkedin/bluepill/pull/376 to back-port the changes to v4.1.1+packing if that is still needed.

[Ashraf-Ali-aa]

Hi is there any update regarding this issue, I am also having this issue, a better way to get test cases is using something like this https://github.com/dropbox/hypershard-ios

[ravimandala]

Thanks for the pointer @Ashraf-Ali-aa. I looked into it and it looks like it works only on Swift test suites and if all test class files are named in a certain pattern. Not sure if we can use it given the limited use cases it serves. I have a solution which is working except for a minor issue in UI tests. I am trying to find some time to complete it.

[Ashraf-Ali-aa]

cool, you can use Idb which I believe it does support objective C and swift https://github.com/facebook/idb

[chenxiao0228]

@ravimandala is this still an issue?