stream_data icon indicating copy to clipboard operation
stream_data copied to clipboard

Shrinking doesn't seem to work as advertised for one_of/frequency/tree

Open obrok opened this issue 7 years ago • 0 comments

I'm using stream_data to generate a rather large data structure representing an SQL query. I was having problems shrinking the generated examples - it seemed like only minor details like integer values were being shrunk while the query structure as a whole remained the same. I managed to reduce it to the following property (should work after pasting into stream_data tests):

    property "shrinks towards earlier generators" do
      check all size <- positive_integer(),
                seed <- {integer(), integer(), integer()} do
        {:error, result} =
          one_of([:foo, {:bar, integer()}])
          |> check_all(
            [max_shrinking_steps: 100, initial_size: size, initial_seed: seed],
            fn example -> {:error, example} end
          )

        assert result.nodes_visited < 100
        assert :foo = result.shrunk_failure
      end
    end

Error:

  1) property one_of/1 shrinks towards earlier generators (StreamDataTest)
     test/stream_data_test.exs:199
     Failed with generated values (after 5 successful runs):

         * Clause:    size <- positive_integer()
           Generated: 4

         * Clause:    seed <- {integer(), integer(), integer()}
           Generated: {3, 0, 0}

     match (=) failed
     code:  assert :foo = result.shrunk_failure()
     right: {:bar, 0}

Curiously - it does seem to be dependent on the size, even though integer/1 which is used by one_of seems to operate independently of size. If I just set size <- constant(1), then the property passes.

I did some tracing and it does seem like for small initial_size the initial failure found is {:bar, _} and after some shrinking it gets reduced to :foo but that doesn't seem to happen for a larger initial_size. It doesn't seem to happen when the generator tested is just one_of([:foo, :bar]).

This also affects frequency/1 as it has almost the same implementation with the minor difference of how the option is chosen from the list and at the very least tree/2, which is how I originally found it. I assume it also affects anything else that uses frequency/1.

I'd be happy to make some more attempts at fixing this, but I haven't yet understood the code dealing with shrinking fully, so I wanted to make sure I'm on the right track here and also ask for any advice you might have.

obrok avatar Apr 13 '18 17:04 obrok