open5e
Check if API Updated
Hello!
Is there a way to tell weather or not the API has been updated? I do not see any sort of version string within the hierarchy of the API data. I apologize if there is already and obvious answer to this question, and I am just missing it.
For more context, I am writing a desktop app (incomingstick/OpenRPG-App) and store the API data locally. I would like to have a way to check whether or not the local data needs to be updated from the API, ideally without needing to pull the entire tree, and without needing to check against GitHub or a git repository. For now I am just running a function to pull the entire data tree from your API on the app start, but that is less than ideal.
I am willing to implement this if you have thoughts on what type of data versioning you would like to see used.
Hi! @eepMoody should certainly weigh in on this but I would like to bring up two things:
- The version of the method used to communicate the data is one thing. (let's say we want to use graphql for our next version!)
- The version of each data item is another (it we could have had a typo, or there could be a WoTC errata on Duergar for example.
- Whether or not the API has added new elements since last time you checked is another.
I've discussed versioning the data separately from the API itself (in a different code repository even). That could be useful. Check the /data folder for where our raw data comes from before we load it. A distinct version-controlled datastore, then set up the API to load data from a branch in that data store, that could be a good architecture.
That's a really interesting idea, one I hadn't considered! Like August is saying, we could probably tackle it at a few different levels.
If we're just looking for a way to know IF the data has updated, we could include a hash of some kind in the list responses?
In the meantime, you could probably use the for dockerfile for the backend as a proxy. Since any change to the data would force a new dockerfile, you can use the current version hash to determine if there's been a change. https://hub.docker.com/r/eepmoody/open5e-beta
Perhaps there's a way to hit that API for now, and if you have ideas on how to implement this I'm open to just about anything. :)
I can work around whatever design choice you think would be best! That said, I would like to provide some of my thoughts on what I would find easiest for my projects workflow
One thought I had was that each file within the data of the API would have its own hash, including the root of the tree. This would then trigger a traversal of the API data tree, where each JSON file would be checked for its hash, and if the hash is different, we would then download the new file. This does still require a full traversal of the API to check for updates.
Another thought, and more ideally, there would be a separate manafest.json, similar to the root, but instead it contains the current "version" hash, and then a list of those files that changed. This feels like the least cumbersome route for my project, as it would make checking for updates the most efficient. I would simply check the hash of my current manifest against the API, and if the hashes differ, we parse the rest of the manifest and only download those files listed within.
Alternatively, my project does use NodeJS (npm) and seeing this as an includable npm package would be also be cool, though I would ideally like to find a way to have this data update independent of OpenRPG, so that I don't have to ship data locked versions of the software.
I am open to any and all ideas, and will conform to whatever method you decide to use!
I think that using Git to version the raw data makes sense if possible.
I’m imagining that the initialization process of the api Would be that it actually reaches out to a preconfigured url (On github) to download the data.
But that doesn’t really solve your problem exactly. (Although hashes could be generated at load time)
Do you know enough python to work with me on some of this?
On Apr 28, 2020, at 6:32 PM, Nick Gaulke [email protected] wrote:
I can work around whatever design choice you think would be best! That said, I would like to provide some of my thoughts on what I would find easiest for my projects workflow
One thought I had was that each file within the data of the API would have its own hash, including the root of the tree. This would then trigger a traversal of the API data tree, where each JSON file would be checked for its hash, and if the hash is different, we would then download the new file. This does still require a full traversal of the API to check for updates.
Another though would be a separate manafest.json, similar to the root, but instead it contains the current "version" hash, and then a list of those files that changed. This feels like the least cumbersome route for )my) project, as it would make checking for updates the most efficient. I would simply check the hash of my current manifest against the API, and if the hashes differ, we parse the rest of the manifest and only download those files listed within.
Alternatively, my project does use NodeJS (npm) and seeing this as an includable npm package would be also be cool, though I would ideally like to find a way to have this data update independent of OpenRPG, so that I don't have to ship data locked versions of the software.
I am open to any and all ideas, and will conform to whatever method you decide to use!
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.
Unfortunately I have only done some very basic Python work, just enough to understand the general syntax of the language. I have worked through roughly half of "Grey Hat Python" by Justin Seitz. Though I feel I have a general enough understanding of most languages to be able to dive right in.
I'll see if I can find time this weekend to fork the repo and start poking around!
I think that this issue is resolved by https://github.com/open5e/open5e-api/pull/76 and can be closed.
