intelowlproject
Allow editing plugin `params` from the GUI
At the moment, the API keys can be inserted only before starting the project, as environment variables.
It would be useful to let the user set up those keys and the other required configuration directly from the GUI.
This would imply a lot of changes. Also, we have already planned to work on the configuration (#306) so this idea could be considered as a later development
After much deliberation/pain, we dropped the idea of storing the analyzer_config,json in the database.
~Now we need another way to handle this. Ideally, a robust way to publish edits/writes directly to the JSON file.~
Why was the idea of storing analyzer_config.json in the database dropped? What exactly kept going wrong? it seems like a feasible thing to do in theory.
The discussion was complex and I think it is forgotten between PRs and slack....this is what I remember right now, but maybe @eshaan7 has something else to say about:
The analyzer_config.json can change over time. This is normal and can happen several times. If we preload this to the database, then, with a new official version of that JSON, we would need to add a way to align the already existing databases for the ones who already have a working IntelOwl instance. But that adds another issue: if the user of that DB has made personal changes, and that is normal, we won't be able to overwrite the database with the new configuration by default. This problem is also available right now if you locally change the analyzer_config.json and add your own analyzers / modify the existing ones. Anyway, imho, this is more managable cause you just need to change a single file and don't mess with the DB.
API keys have not this problem so, for this case, it makes sense to store them into the database and have them managable from the GUI. It would be great to have a "marketplace" where it would be possible to check the analyzers/connectors configuration and insert the keys directly there. This would be super useful.
Then, I also think that the analyzer_config.json file has become too much big and we should start to think to handle this differently (by moving all that JSON code to python. This could be another refactor issue)
With a JSON file, I feel like it's pretty flexible for development and on-the-go changes, but as the project grows the JSON is going to get highly populated. I think moving to a python based config seems suitable. This can be introduced with a major version change like dev-v4 GUI.
Another thing that came to my mind while reading the issue was that we could also make use of a quick converter, a startup script that converts JSON to a DB / config file to maintain backward compatibility.
a startup script that converts JSON to a DB / config file
This was actually one of the original ideas - load the JSON into DB as django models and allow read/writes to these models from the django admin and a DRF view for the GUI.
Only issue is this should be accomplished while keeping the JSON file still as the single source of truth while storing the user modified changes in the DB. This means user should not need to touch anymore the JSON files and only use the django admin / React GUI to update the configurations.
Hopefully I should find sometime during the weekend/next week where I can implement a MVP of this idea.
So, initially we convert the JSON configuration to a DB, then allow changes to the DB from the GUI, instead of editing JSON everytime. Sounds neat, that way the JSON can function like a skeleton config for the DB.
With due respect Sirs, I'm somewhat interested in the issue and was thinking if allowed, I would like to work on this for GSOC. I haven't yet thought of a proposal yet, so I would really appreciate both of yours suggestions on this :)
Hello Mr. @eshaan7 and Mr. @mlodic I feel interested in working on this issue for GSoC 2022. Can I please takeover this issue for the proposal?
Rough explanation of a solution I had in mind:
We could replace the dataclasses with django models like so:
class Plugin(models.Model): # applies to both Analyzer and Connector
name = models.CharField(primary_key=True)
params = models.JsonField()
This way we can allow users to store their own versions of config dict against a plugin name. Rest of the fields maybe do not need to duplicated in the DB and can be mapped (mapped and read from the config serializers) using @property decorators.
Thank you very much for the heads up Eshan Sir. I'm currently going through the Django sources thoroughly while also learning more about Django. I'll try my best to present a draft soon, before the proposal is ready if possible :)
This issue was tracking 2 different feature requests together which was creating confusion so I splitted it into 2 issues as these can be independently worked upon:
- #433
- #978