Don't mutate and test all in one go

[dnlserrano]

More context on this original post over at ElixirForum.com where I first explained it.

Right now I’m not running each test ... do individually but instead I’m running the whole test module (e.g., HelloWorldTest). This has one clear disadvantage, which I’ll explain below with an example:

defmodule HelloWorld do
  def sum(a, b) do: a + b
  def divide(a, b), do: div(a, b)
end

defmodule HelloWorldTest do
  test "when testing sum" do
    assert HelloWorld.sum(3, 0) == 3
  end

  test "when testing divide" do
    assert HelloWorld.divide(5, 2) == 3
  end
end

If I change code to the following:

defmodule HelloWorld do
  def sum(a, b) do: a - b # changed from + to - via AOR1
  def divide(a, b), do: div(a, b)
end

I will be running the tests for both tests, instead of just running the test for sum/2 (i.e., "when testing sum", which was the only one for which the corresponding source code changed). In order to try and maximise the amount of mutations I can catch with running the entire test module, I mutate all in one go. Does that make sense? Maybe it doesn’t… 🤦‍♂️ AFAIU, finding out what tests I should run per source code change is hard (or at least not trivial? 😅). But I might not be seeing something very obvious.

Hope people can help out with ideas for this one. Maybe traversing tests and annotating to keep track of what operators are present in each and hence are influenced/swing based on mutations performed? 🤔