Custom Directives and Example: Integration with MetaFlow

[jimmiemunyi]

Tagging @hamelsmu since we had talked about this earlier.

Description

I was working on a custom use of Nbdev, using it to define my DAGs. This is because I usually prototype in a notebook and then copy the important code when writing my DAGs in my IDE anyway.

Here is an example repo, where I prototype training MNIST with fastai on a notebook and attempt to create the DAG Workflow from the notebook: https://github.com/jimmiemunyi/nbdev-metaflow-example. In the end, I was able to achieve the desired result, but with some hiccups here and there. The notebooks are in the nbs folder and the generated code is in the nbdev_metaflow_example/workflows folder.

I have also highlighted issues I came across in the notebooks but will also highlight them here in this issue

Issues I encountered.

• If we can have custom Directives, we can potentially adapt nbdev to particular use cases when developing code. People could even write extensions to nbdev for specific libraries like nbdev_metaflow_extension. An example of working with Metaflow :
  • we do not require the __all__ at the top since the WorkFlows are not meant to be imported. If one could write a custom directive instead of using the default #| default_exp foo, I would probably customize it to fit the way Metaflow expects the final file to be.
  • we need an __if__ == '__main__' at the end of every file to call the workflow. I had to manually add it all the time

if __name__ == '__main__':
    SimpleMNISTFlow()

• Monkey patching but with passing in a decorator. Suppose I want to monkey patch a method into a class, but the method also requires a decorator, how would I achieve this? This is highlighted in the 01_Complex_MNIST_Example.ipynb Example:

    @conda(libraries={'numpy':'1.18.1'})
    @resources(memory=1000)
    @step
    def start(self):
        """
        Parse the MNIST Dataset into Flattened and None Flattened Data artifacts. 
        """
        import numpy as np
        
        self.data = np.array([2.0, 3.0])

The patch functionality comes from fastcore so this is a big if. I am not sure if it's possible or if it would not be practical:

@patch
def _dummy(self:ComplexMNISTFlow, decorators={'resources': 'memory=1000'}):
    pass

Anyway, big fan of the library and I use it almost all the time I want to develop some python code.