Are there any demos?

[Urheen]

Could you please provide any demos for using this package?

[janEbert]

Oh sorry, the README is kind of incorrect here, I have to fix that! In fact, you have to pass a function to the VeLO.step method that

1. takes no arguments,
2. calculates gradients for the model, and
3. returns the loss.

This can be a bit awkward to achieve in a multi-processing-safe way.

The usage section in the README should be something like:

from pytorch_velo import VeLO

# [...]

train_steps = epochs * len(dataset)  # Assuming `dataset` is already batched.
opt = VeLO(params, num_training_steps=train_steps, weight_decay=0.0)

def loss_with_backward_builder(inputs, targets):

    def loss_with_backward():
        opt.zero_grad()
        preds = model(inputs)
        loss = criterion(preds, targets)
        loss.backward()
        return loss

    return loss_with_backward

# [...]

# For each batch of input-target pairs:
closure = loss_with_backward_builder(inputs, targets)
opt.step(closure)

[Urheen]

Thank you very much for your reply!

Could you please add one attribute that it can load the pretrained model of velo optimizer?