Potential issue with shrinking when combined with scaling

[keathley]

I'm not sure if the title is correct or makes sense. Generally I'm having a hard time getting reasonable counter-examples.

I have some code that needs to filter a list of elements based on a provided whitelist. The test for this looks like this:

defmodule MyTest do
  def good_fields(whitelist) do
    sized fn size ->
      whitelist
      |> one_of
      |> list_of(min_length: size)
      |> map(fn list -> Enum.map(list, &Atom.to_string/1) end)
    end
  end

  def whitelist do
    bind scale(list_of(atom(:alphanumeric), min_length: 1), & &1*3), fn list ->
      constant(Enum.uniq(list))
    end
  end

  describe "whitelist/2" do
    property "returns only the whitelisted fields" do
      check all \
        whitelist <- whitelist(),
        real      <- good_fields(whitelist),
        garbage   <- scale(list_of(binary()), & &1*4)
      do
        fields = real ++ garbage
        assert FieldSelector.whitelist(fields, whitelist) == real
      end
    end
  end
end

The actual issue is that I didn't ensure that the garbage fields were not included in the whitelist. StreamData wouldn't find this issue until I scaled both the whitelist generator and the garbage generator. Once I had done that I got this counter example:

  1) property whitelist/2 returns only the whitelisted fields
     Failed with generated values (after 24 successful runs):
     
         * Clause:    whitelist <- whitelist()
           Generated: [:f3s, :mehtwbp, :aRknJKop0hFvA873k1@of3, :P5U_VEToigea5IhP, :_mEylhu, :I, :gTj2sLvk8DSf@Y, :pEpfY87ZPWoc1nMgUK_s0F, :fNHuckgqQmLfJhb32k3dg, :nFX, :szyUhwHfrSF0jX_03qNr@PxLlj, :oz, :e5yQ5EQx2M87M13EwpSLktAwQ, :jZiDjaKT@, :pSxD4DIy7yrMGJNz, :mH9O_XKAg4@0ES6bEy65NF2H, :_li9HD5zl5qU4SN, :qGOGFYfuhkWxuSww, :ebKfOTMcgOom0Yx01bC8e1az, :_LgHwcU, :VsZfa4J, :_IjCozJ9ZpBTHulF, :_, :JYf, :vWX4f2, :_FjJSsaxq456CMRxXBVA@, :_EKG9NLyw103Ft0T0, :Nxp2WmQRSmCEQM, :_j6A, :a, :Qa@AtebCMFv_FYSbE7wooT52, :w@fl6vk6Y1, :lR83aOqI, :pVTD@@I, :g5cj8Cs6rIJ5KJnV0b, :f8lJjMY6YjhCzpy@0l, :ig, :h87XKO3E9DIOL@nn22I, :AUgq, :nhGolyOG_Tt6sonFAFw7a6mrC, :TN78mT0JcFD@vkPQMl, :kGwaw9y, :_19, :xIZEIm, :O7@g0ceYE, :tjzM, :nHIsiP5RW54r, :aC8AOE_AsOCiMC, :_E5DlrUqs8mcSWyjYerfkBQLPN]
         
         * Clause:    real <- good_fields(whitelist)
           Generated: ["_j6A", "nHIsiP5RW54r", "TN78mT0JcFD@vkPQMl", "_FjJSsaxq456CMRxXBVA@", "tjzM", "_j6A", "pVTD@@I", "_FjJSsaxq456CMRxXBVA@", "f3s", "nhGolyOG_Tt6sonFAFw7a6mrC", "_LgHwcU", "pVTD@@I", "_li9HD5zl5qU4SN", "mH9O_XKAg4@0ES6bEy65NF2H", "O7@g0ceYE", "P5U_VEToigea5IhP", "_mEylhu", "f3s", "pSxD4DIy7yrMGJNz", "pSxD4DIy7yrMGJNz", "aC8AOE_AsOCiMC", "nhGolyOG_Tt6sonFAFw7a6mrC", "VsZfa4J", "aRknJKop0hFvA873k1@of3", "_LgHwcU"]
         
         * Clause:    garbage <- scale(list_of(binary()), &(&1 * 4))
           Generated: ["I", <<122, 255, 43, 70, 243, 198, 209, 254, 82, 204, 29, 85, 140, 141, 191, 173, 159, 138, 168, 41, 39, 114, 95, 97, 38, 109, 174, 9, 83, 228, 155, 4, 169, 188, 249, 10, 26, 207, 141, 186, 88, 164, 136, 233, 63, 190, 206, 136, ...>>, <<131, 74, 242, 191, 247, 124, 229, 255, 214, 174, 143, 28, 146, 29, 88, 80, 53, 34, 177, 251, 31, 62, 204, 6, 62, 24, 153, 239, 151, 142, 225, 96, 217, 244, 138, 78, 136, 7, 92, 176>>, <<36, 176, 143>>, <<126, 166, 52, 245, 162, 110, 203, 181, 235, 11, 92, 252, 183, 115, 198, 191, 3, 160, 203, 159, 43, 220, 37, 73, 134, 163, 241, 204, 18, 234, 76, 169, 25, 176, 122, 107, 51, 216, 28, 157, 191, 43, 43, 245, 233, ...>>, <<105, 89, 96, 5, 54, 36, 29, 169, 245, 241, 43, 211, 107, 115, 107, 200, 77, 96, 106, 159, 153, 70, 193, 190, 124, 133, 2, 224, 16, 33, 95, 249, 238, 153, 143, 100, 49, 241, 186, 165, 35, 172, 253, 103, ...>>]
     
     Assertion with == failed
     code:  assert FieldSelector.whitelist(fields, whitelist) == real
     left:  ["_j6A", "nHIsiP5RW54r", "TN78mT0JcFD@vkPQMl", "_FjJSsaxq456CMRxXBVA@", "tjzM", "_j6A", "pVTD@@I", "_FjJSsaxq456CMRxXBVA@", "f3s", "nhGolyOG_Tt6sonFAFw7a6mrC", "_LgHwcU", "pVTD@@I", "_li9HD5zl5qU4SN", "mH9O_XKAg4@0ES6bEy65NF2H", "O7@g0ceYE", "P5U_VEToigea5IhP", "_mEylhu", "f3s", "pSxD4DIy7yrMGJNz", "pSxD4DIy7yrMGJNz", "aC8AOE_AsOCiMC", "nhGolyOG_Tt6sonFAFw7a6mrC", "VsZfa4J", "aRknJKop0hFvA873k1@of3", "_LgHwcU", "I"]
     right: ["_j6A", "nHIsiP5RW54r", "TN78mT0JcFD@vkPQMl", "_FjJSsaxq456CMRxXBVA@", "tjzM", "_j6A", "pVTD@@I", "_FjJSsaxq456CMRxXBVA@", "f3s", "nhGolyOG_Tt6sonFAFw7a6mrC", "_LgHwcU", "pVTD@@I", "_li9HD5zl5qU4SN", "mH9O_XKAg4@0ES6bEy65NF2H", "O7@g0ceYE", "P5U_VEToigea5IhP", "_mEylhu", "f3s", "pSxD4DIy7yrMGJNz", "pSxD4DIy7yrMGJNz", "aC8AOE_AsOCiMC", "nhGolyOG_Tt6sonFAFw7a6mrC", "VsZfa4J", "aRknJKop0hFvA873k1@of3", "_LgHwcU"]

The minimal counter example is:

whitelist = [:I]
real = []
garbage = ["I"]

It seems like stream_data should be able to further shrink this counter example by quite a bit. Is there some interaction between scaling and shrinking? Is there some other way to improve the shrinking here?