knock

[g4ze]

fixes #1418

Description

Please include a summary of the change and link to the related issue.

Type of change

Please delete options that are not relevant.

• [ ] Bug fix (non-breaking change which fixes an issue).
• [x] New feature (non-breaking change which adds functionality).
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist

• [x] I have read and understood the rules about how to Contribute to this project
• [x] The pull request is for the branch develop
• [x] A new plugin (analyzer, connector, visualizer, playbook, pivot or ingestor) was added or changed, in which case:
  • [x] I strictly followed the documentation "How to create a Plugin"
  • [x] Usage file was updated.
  • [ ] Advanced-Usage was updated (in case the plugin provides additional optional configuration).
  • [ ] I have dumped the configuration from Django Admin using the dumpplugin command and added it in the project as a data migration. ("How to share a plugin with the community")
  • [ ] If a File analyzer was added and it supports a mimetype which is not already supported, you added a sample of that type inside the archive test_files.zip and you added the default tests for that mimetype in test_classes.py.
  • [ ] If you created a new analyzer and it is free (does not require any API key), please add it in the FREE_TO_USE_ANALYZERS playbook by following this guide.
  • [ ] Check if it could make sense to add that analyzer/connector to other freely available playbooks.
  • [x] I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
  • [ ] If the plugin interacts with an external service, I have created an attribute called precisely url that contains this information. This is required for Health Checks.
  • [x] If the plugin requires mocked testing, _monkeypatch() was used in its class to apply the necessary decorators.
  • [x] I have added that raw JSON sample to the MockUpResponse of the _monkeypatch() method. This serves us to provide a valid sample for testing.
• [x] If external libraries/packages with restrictive licenses were used, they were added in the Legal Notice section.
• [x] Linters (Black, Flake, Isort) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
• [ ] I have added tests for the feature/bug I solved (see tests folder). All the tests (new and old ones) gave 0 errors.
• [ ] If changes were made to an existing model/serializer/view, the docs were updated and regenerated (check CONTRIBUTE.md).
• [ ] If the GUI has been modified:
  • [ ] I have a provided a screenshot of the result in the PR.
  • [ ] I have created new frontend tests for the new component or updated existing ones.
• [x] After you had submitted the PR, if DeepSource, Django Doctors or other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.

Important Rules

• If you miss to compile the Checklist properly, your PR won't be reviewed by the maintainers.
• Everytime you make changes to the PR and you think the work is done, you should explicitly ask for a review. After being reviewed and received a "change request", you should explicitly ask for a review again once you have made the requested changes.

{ "marcia.domain.com": ["66.96.162.92"], "http": [404, None, "Apache"], "https": [None, None, None], "cert": [None, None], }, { "mbsizer.domain.com": ["66.96.162.92"], "http": [404, None, "Apache"], "https": [None, None, None], "cert": [None, None], }, { "malektravel.domain.com": ["66.96.162.92"], "http": [403, None, "Apache"], "https": [403, None, "Apache"], "cert": [True, "2024-10-08"], }, { "mchattan01.domain.com": ["66.96.162.92"], "http": [200, None, "Apache"], "https": [None, None, None], "cert": [None, None], }, { "martina6marco.domain.com": ["66.96.162.92"], "http": [500, None, "Apache"], "https": [None, None, None], "cert": [None, None], }, { "maludomaincom.domain.com": ["66.96.162.92"], "http": [403, None, "Apache"], "https": [403, None, "Apache"], "cert": [True, "2024-10-08"], }, { "margaretlion.domain.com": ["66.96.162.92"], "http": [200, None, "Apache"], "https": [200, None, "Apache"], "cert": [True, "2024-10-08"], },

[g4ze]

@mlodic is there a way to safelt remove the file generated after the scan. I'm not able to find a way to specify the directory where the file is being written to. It is by default being saved in the project's main directory.

[mlodic]

there's the json option as output, have you tried it?

[g4ze]

Okay, i read the knockpy module again, the way i have written the code shouldn't exactly produce any file. I just ran the analyzer after deleting the file and it seems to work now. No file is being produced so deleting it is no more a problem anymore. My bad for the incontinence caused! pushing the commit

[g4ze]

will add playbook migration in a while

[mlodic]

playbook migration could be avoided cause this is a scanner and we don't want people to accidentally execute it against sites

[g4ze]

the tests always timeout. maybe because the value is set to be 20s max and knock analyzer takes a lot of time to process.

[0ssigeno]

Mmmh, I do not like to increase the timeout because the CI is already slow as fuck, but I don't see other way then to increase it, if we do not want to mock it up. I would suggest these changes: tests/api_app/analyzers_manager/test_classes.py:112 from

                timeout_seconds = min(timeout_seconds, 30)

to

                timeout_seconds = max(timeout_seconds, 30)

And then change the plugin soft_time_limit to a value that allows its execution (you must do it in the migration file that you made).

[g4ze]

Why don't we try and mock it up as 600s would be a lot? How is that supposed to function? Also, true, CI is slow af, is there a way to not rebuild every time until docker stuff is changed and make the setup faster? A way to only run the test for the analyzer that just changed. Maybe we can have two types of workflows? One for quicker development and one before a final merger of the branch?

[0ssigeno]

I don't get the first question. if you think that the analyzer requires 10 minutes to function, you can make the changes that I added before, and set the soft_time_limit at 600 and should work.

As for the CI, unfortunately we don't have the time to improve it. You can always run the single tests that is failing in your local environment, and check the result before uploading it.

[g4ze]

Understood!

[g4ze]

even putting t/o for 600s didn't get the job done. i think its better to mock response in the test or something else... I can't seem to get that done as well.

[g4ze]

I'll have to look into the mocking function carefully, something is wrong there...

[mlodic]

I don't want to ruin the party cause the analyzer is ok but this line is not

git+https://github.com/guelfoweb/knock.git

We noticed that this thing would force IntelOwl to rebuild again and again the package every time either a CI action is triggered or you are rebuilding locally that package. That makes the build really slow. We would need to leverage a pre-build package published in pypi. I have created an issue about that because we have the same problem with oletools now (https://github.com/intelowlproject/IntelOwl/issues/2347).

I know the authors of those tools and I have just contacted them about solving this issue. I don't want to keep this issue open forever so I would like to just wait for few days to see if the author answers or not so we can fix the problem in advance

[g4ze]

😭😭

[mlodic]

ref: https://github.com/guelfoweb/knock/pull/148

[mlodic]

I am in contact with the author, at end of June will get back to me for the release of an official version. If we don't get it in time, we would need to fork it and create our own package

[mlodic]

knockpy-subdomains is available!!!!

[g4ze]

Woohoo!! Going back to college, a lot of shifting(again), will finish this one till weekend most probably.

[g4ze]

Hey! @mlodic I am facing the same issue, the one i faced in #2431 , I've implemented the library. Could you please pull and run this on your local intelowl. There might be some problem on my side.

[mlodic]

please open a new PR in the IntelOwl repo so it's easier to collaborate (from a branch in IntelOwl main repo)

[g4ze]

New PR at #2448