mitre
Fix - Broken tests
Description
Fixes a broken test.
Type of change
Please delete options that are not relevant.
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] This change requires a documentation update
How Has This Been Tested?
I ran all tests, including the fixed one.
Checklist:
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [ ] I have made corresponding changes to the documentation
- [ ] I have added tests that prove my fix is effective or that my feature works
just for my awareness, what issue is this fixing? The current atomic tests seem to work for me locally
@uruwhy, without this change, the test plugins.atomic.tests.test_atomic_svc.TestAtomicSvc.test_use_default_inputs (plugins/atomic/tests/test_atomic_svc.py:181) can fail in certain circumstances.
Summary
It happens when Caldera has already run once with the plugin atomic enabled before running this test. This happens frequently while developing.
plugins/atomic/tests/test_atomic_svc.py:180 (TestAtomicSvc.test_use_default_inputs)
'64c4ae_qakbot.bat -a' != 'PathToAtomicsFolder\\T1016\\src\\qakbot.bat -a'
Expected :'PathToAtomicsFolder\\T1016\\src\\qakbot.bat -a'
Actual :'64c4ae_qakbot.bat -a'
<Click to see difference>
self = <test_atomic_svc.TestAtomicSvc object at 0x10c1d9520>
atomic_svc = <plugins.atomic.app.atomic_svc.AtomicService object at 0x10c1da6f0>
atomic_test = {'auto_generated_guid': '121de5c6-5818-4868-b8a7-8fd07c455c1b', 'description': 'A list of commands known to be perform...omicsFolder\\T1016\\src\\qakbot.bat', 'description': 'File that houses commands to be executed', 'type': 'Path'}}, ...}
def test_use_default_inputs(self, atomic_svc, atomic_test):
platform = 'windows'
string_to_analyze = '#{recon_commands} -a'
got = atomic_svc._use_default_inputs(test=atomic_test,
platform=platform,
string_to_analyse=string_to_analyze)
> assert got[0] == 'PathToAtomicsFolder\\T1016\\src\\qakbot.bat -a'
E AssertionError: assert '64c4ae_qakbot.bat -a' == 'PathToAtomicsFolder\\T1016\\src\\qakbot.bat -a'
E
E - PathToAtomicsFolder\T1016\src\qakbot.bat -a
E + 64c4ae_qakbot.bat -a
plugins/atomic/tests/test_atomic_svc.py:187: AssertionError
Details
The test code first calls plugins.atomic.app.atomic_svc.AtomicService._use_default_inputs, then plugins.atomic.app.atomic_svc.AtomicService._catch_path_to_atomics_folder.
The latter can return 2 different values.
The test success depends on to the returned value.
In the capture below, I called in advance the method in the debugger to see its result before its real call (overriding variables with returned values):
Everything happens in this method.
In the case you start Caldera with the atomic plugin at least once, the plugin, when starting, clones the atomic-red-team repository, and the testing path to the payload plugins/atomic/data/atomic-red-team/atomics/T1016/src/qakbot.bat becomes valid.
The if clause is validated, making a call to plugins.atomic.app.atomic_svc.AtomicService._handle_attachment and changing the resulting payload name (includes a hash)
Therefore, the whole test fails.
Reproducing
To make the test pass, just deletes the generated atomic-red-team repository (plugins/atomic/data/atomic-red-team).
This is the cas after cloning the Caldera repository without the first run.
To make the test fail again, deletes the plugins/atomic/data/abilities repository and start Caldera with the atomic plugin enabled.
The plugin will regenerate the atomic-red-team repository and the abilities directory.
My fix proposal
What I did here is to change the testing default path of the atomic_test fixture (dictionary) to make sure it won't exist, whatever the atomic-red-team repository is cloned or not.
This is one solution I found, but there should be other ways to make this this test deterministic. However, I'm not expert in Caldera and its plugins. I am open to suggestions.