Rework DB backend
There are various problems with the architecture and the use of SQLite.
P1: SQLite is inflexible
The current SQLite database isn't very flexible and requires a lot of work when changing the structure or adding a new field.
Solution: YAML
Change backend and store it as YAML or JSON. This allows for more flexibility when adding or removing a field without throwing an error. This will make it easier to debug or check what's in the DB and remove faulty things as it's more human-readable than SQLite.
P2: Running queries
It's hard to run queries because the data isn't structured very well as there cannot be any hierarchy
Solution: Load into memory and classes
- As the DB will be small, the best would be to load it entirely into memory
- Do all the searching in python without any queries. This makes it easy to follow and test.
P3: How to handle CRUD operations?
How to handle CRUD operations? Will there be a separate repository? Should we go with a different repository?
Solution: ???
P4: Rearchitecture repos?
Should we change how we do repositories? Instead of download_repo, we should inject the different gateways/repositories that are used? Need more thought on this.