Bart van Merriënboer

We call the on-the-fly preprocessors [data streams](http://blocks.readthedocs.org/en/latest/datasets.html), while the datasets themselves are immutable. For a lengthier discussion on how we do checkpointing you can look [here](http://blocks.readthedocs.org/en/latest/serialization.html).

http://deeplearning.net/software/pylearn2/library/devtools.html#module-pylearn2.devtools.nan_guard On Feb 25, 2015 2:40 PM, "Dmitry Bogdanov" [email protected] wrote: > See discussion in #343 https://github.com/bartvm/blocks/pull/343. > > — > Reply to this email directly or view it on...

And if you want to avoid updating instead of stopping, the new plugin by @memimo does that, no? On Feb 25, 2015 2:40 PM, "Dmitry Bogdanov" [email protected] wrote: > See...

The new step rule avoids updating of you set the scaling to 1, so that + NanGuardMode might be enough?

Theoretically, no solution is ever going to be as fast as not checking, right? Even if you could optimize the hell out of it, at the very least you still...

I think we'll have to ask @nouiz about the details, but as far as I can tell from the code, [Pylearn2's NaNGuard](https://github.com/lisa-lab/pylearn2/blob/master/pylearn2/devtools/nan_guard.py) compiles with the same mode, so optimizations should...

Sounds good to me, but just to clarify: This does mean that the user won't be able to retrieve the state of the parameters from right before `NaN` occurred, right?...

Yeah, that sounds easy enough, just do something like `tensor.any([tensor.any(tensor.or_(tensor.isnan(step), tensor.isinf(step)) for step in steps]) -> scale all parameters` (with 1.0 potentially)

I would have to dig up the old e-mails, but I'm not too enthusiastic. @rizar's comment already touches on what I think would be problematic, each `_push_initialization_config` method would end...

How would we support the `force` operator then? Not allowing overriding could be done by just taking the attributes from the `lazy` decorator and turning them into properties somehow that...