Deep-Embedded-Validation icon indicating copy to clipboard operation
Deep-Embedded-Validation copied to clipboard

get_dev_risk returns NaN and very large negative values

Open trivecitoni opened this issue 4 years ago • 1 comments

There doesn't seem to be any bound on the value returned by get_dev_risk.

For example:

N = 1000
weight = np.ones((N,1))
error = np.random.rand(N,1)
get_dev_risk(weight, error) 
# outputs nan

I can obtain very large negative values as well:

N = 1000
np.random.seed(6382)
weight = np.random.normal(loc=0.5, scale=0.000001, size=(N,1))
error = np.random.rand(N,1)
get_dev_risk(weight, error)
# outputs -8220.64

Is this the expected behavior?

trivecitoni avatar Aug 27 '21 05:08 trivecitoni

Hi, sorry for the late reply (I forgot to set the notification of this repo =_=|| ). The code assumes weight is a random variable with expectation of 1, and the variance of weight is used for the control variable to reduce the variance. You can check the paper for details.

In your first code, weight = np.ones((N,1)) results in NaN because the variance is 0, so this line would not work.

In your second code, weight = np.random.normal(loc=0.5, scale=0.000001, size=(N,1)) does not obey the requirement that its expectation should be 1.

Actually the weight should come from the dev.py:get_weight function and it is not arbitrary. Therefore we can guarantee meaningful results.

youkaichao avatar Oct 01 '21 15:10 youkaichao