pyvec
Split data fetching and building static HTML into 2 commands
On slow connections (like mine) the first request to python runserver.py serve takes
e
t
e
r
n
i
t
y
.
It's because the talks archive repo is cloning and basically everything is building. Maybe this could be happening only before the first request to the relevant routes? 🤔 The same could be true for all other requests doing significant computation.
This is something I was able to address with a new architecture in my experimental "fork" - https://github.com/honzajavorek/py If the "fork" becomes upstream one day, this issue can be closed.
@honzajavorek the fork is now archived - was this issue solved and can be closed, or needs to remain open and we can add some acceptance criteria how much time on what computer will be considered as solved..?
I don't think it has been solved.
The fork is discontinued, but I've tested some ideas there and then used them when starting the Frozen-Flask app powering junior.guru. As of 2022 I'd rather describe the ideas and lessons learned, than look back to any code in that fork.
The basic idea is that the app has two separate commands for:
- syncing (fetching) the data from various sources to an SQLite db
- serving the site locally or building the HTML of the site from the SQLite db
Thanks to such architecture it's possible to sync the data once (in a while), then serve the site locally and quickly iterate when developing. It's possible to easily back up the production data by zipping and saving a single .db file. Such architecture also helps to separate resposibilities of the code. The web rendering code doesn't need to know anything about the data sync, it just reads the SQLite and renders HTML, simple. The sync code can be convoluted, but is easier to handle as a set of separate scripts, each doing a single job. It's then also easier to split the sync scripts apart and e.g. run them in parallel or in separate CI jobs, if needed.
On a small site like python.cz or pyvec.org this shouldn't be so hard to achieve. I currently can't think of a more fitting architecture for the pattern we're trying to tackle here, i.e. getting some data from external or local sources, then rendering a website of static HTML.
I don't have permission to rename issues => I'd suggest Split data fetching and building static HTML into 2 commands to make it more clear what is the goal of this task for contributors not familiar with project history.