jupyter-book Can I use jupyter-book as a lab notebook?

I'm stumbled across jupyter-book today as I'm looking to improve my electronic lab notebook. Does anyone do this already? Is this a good/bad idea? It seems like it might work, but I'm not entirely sure. Right now I have several years of my lab notebook in LaTeX, but it is starting to show it's age. I can write some Python to do most of the converting, once I better understand the contents of jupyter-book.

Some things that are important for my interests:

Dated entries, almost like a blog
Ability to add multiple tags to an entry

I'm also intrigued by the ability to include executable code in each entry. That could be particularly interesting.

I hope this is the right place to ask such a question. If it isn't, please let me know where would be better.

Jun 26 '20 15:06 jlconlin

This is a really interesting idea - I hadn't really thought about it before, but I bet that it could be used for something like this. When you say "dated entries" what exactly does that mean? My thinking is that you could create a folder/sub-folder structure for year/month and within each have a collection of posts in that month. For "tags" is this for searchability?

Either way, I'd love to see you give it a shot and then report back where the pain points are :-)

Jun 27 '20 00:06 choldgraf

We do something like this for our quiz question banks, using cell metadata to sort questions/answers, course etc. and it works well. Some simple filter examples for myst:md here.

Two things you may already know:

being able to use ripgrep to recursively search the markdown files is very useful.
pandoc plus jupytext is great for moving latex to myst. It took me about 4 hours to process 17,000 lines of latex for the (in-progress) port of How to think like a computer scientist

As a long-time emacs user, I'll add that what you're really looking for is org-mode -- but the learning curve is not gentle.

Jun 27 '20 23:06 phaustin

@choldgraf Currently, I create a new entry every day in my LaTeX notebook. That way things are dated so I know when things happened. I want to keep this as the main structure, but add tags so I can filter all entries based on those tags. The tags would be associated with particular tasks that I did that day.

Jun 29 '20 15:06 jlconlin

well just a note that if you're writing in markdown for Jupyter Book, you can include any kind of top-level metadata for the page (key: val pairs). Though right now I don't believe we do anything with dates etc in Jupyter Book, at least they'll be there?

Jun 29 '20 15:06 choldgraf

Jumping in to add my support for electronic lab notebooks (ELNs) as a great use case that is covered by Jupyter Book! Thank you to the devs for a really fantastic package, it's been a blast trying it out. Also, apologies for the long post, I can delete and share elsewhere if this is not the place for it.

I saw in another thread that the devs are interested in use cases, so here is mine: I'm a scientist who does both wet lab (biology experiments) and dry lab (molecular modeling and data science), and many of the ELNs used in wet lab science don't have good integration for the computational parts of projects. For me, the most important parts of my ELN are: easy to edit with text and the very occasional image inserted, good published format for personal viewing and sharing, version-controlled, and ability to mix code execution with plain text (for initial data processing next to documentation of data collection).

I currently use markdown files for my daily experimental documentation, and scripts and notebooks (.ipynb and previously .Rmarkdown) for my data analysis. I don't do my note-taking in Jupyter notebooks because markdown is more readable/portable. I was on the cusp of trying to patch together my own pipeline of Jupyter notebooks plus jupytext plus a static site generator, but then I discovered Jupyter Book!

I totally agree with @phaustin that org-mode covers the ELN use case well but the learning curve is high. I've also had some success setting up an ELN with the bookdown package in R, using .Rmarkdown files. Ultimately, the ease of set-up, the fluidity of the Sphinx build process as currently implemented by Jupyter Book, and the ease of kernel switching for .ipynb and MyST Markdown documents have convinced me on Jupyter Book.

I've been testing Jupyter Book (on Mac OS) and everything works great so far. Some things that I've noticed:

I was worried about the number of files getting very large very fast, with one file per experiment. Fortunately, the jupyter-book toc mybookpath/command handles everything exceptionally well (as long as there's a markdown file per folder). One improvement could be to implement a way to build all files in a desired directory instead of listing them one-by-one in the _toc.yml to keep _toc.yml from getting too big, but that's a minor concern for me personally, especially since the toc sidebar has section collapsing. I saw a wild card solution was suggested in the blog thread https://github.com/executablebooks/jupyter-book/issues/900#issuecomment-688251127. The bookdown package's solution was similar, providing a config setting for recursively searching directories and building all files labeled .Rmd (rmd_subdir, at this doc), but I can understand if Sphinx's build process makes this approach less applicable for Jupyter Books, even with the exclude_pattern functionality.
One thing I've had trouble with in other static site generators is handling of relative paths within code executables. This functionality is necessary for those who store data in a separate folder for use by multiple scripts/notebooks. The relative paths have been working great and I have a very functional notebook made from this basic structure:

myproject |── data |── notebook └── experiments └── experiments.md └── experiment_type1 └── YYYYMMDD_description.md <-- relative path uses files in 'data' └── experiment_type2 └── YYYYMMDD_description.md └── index.md └── _config.yml └── _toc.yml └── logo.png └── references.bib └── requirements.txt |── other_directories (folders of PDFs, presentations, etc)

Once again, thanks to all of the contributors to the project! For me, Jupyter Book has provided the best working ELN from markdown files yet.

Sep 17 '20 23:09 cjmathy

Brilliant thanks @cjmathy, so the take-away from your thoughtful comment is that jupyter-book is perfect lol, except for wild cards in the toc (but any thoughts/suggestions keep them coming 😄).

I think at least for non-recursive folder content, toc wildcards should be very achievable (you can already do it in normal sphinx toctree directives) and will pull this out into a separate issue.

(as mentioned this is also related to https://github.com/executablebooks/jupyter-book/issues/900#issuecomment-688251127)

Sep 17 '20 23:09 chrisjsewell

The learning curve for the Emacs is steep. The learning curve for org-mode is not so bad if you limit your focus to mastering the markup language and its support for literate programming. Org-mode is between that for Markdown and LaTeX. Org-mode becomes overwhelming when you try to simultaneously master org-agenda.

The downside of Emacs is that it cannot easily display large images in the org document in an interactive session, although they can be exported to HTML or PDF. You have to link to the external image file and display it in a separate buffer.

Jupyter notebooks are much better at displaying large images and are able to embed them, thereby easing the sharing of the notebook by e-mail by reducing the number of files to send.

Aug 01 '23 10:08 MooersLab