Make replace boolean `check_inferred` with something that allows configuable small unions

[oxinabox]

occured to me during https://github.com/JuliaDiff/ChainRules.jl/pull/717 that there are like 3 states we really care about:

1. Doesn't infer at all, so infers to Any or some UnionAll or abstract type
2. Infers down to a small union of <4 types, so the small union optimization kicks in
3. Infers down to a single concrete type

The current check_inferred boolean doesn't allow us to distinguish case 1 from case 2.