magika icon indicating copy to clipboard operation
magika copied to clipboard
verified publisher icon google

Test support for Windows

Open reyammer opened this issue 1 year ago • 3 comments

reyammer avatar Feb 16 '24 11:02 reyammer

Running the tests for Magika on a Windows platform with py3.11:

============================= test session starts =============================
platform win32 -- Python 3.11.8, pytest-7.4.0, pluggy-1.2.0
rootdir: D:\a\magika\magika\python
configfile: pytest.ini
collected 41 items / 2 deselected / 39 selected

tests/test_magika_python_cli.py::test_magika_python_cli_with_one_test_file FAILED [  2%]
tests/test_magika_python_cli.py::test_magika_python_cli_with_very_small_test_files FAILED [  5%]
tests/test_magika_python_cli.py::test_magika_cli_with_small_test_files FAILED [  7%]
tests/test_magika_python_cli.py::test_magika_cli_with_empty_file FAILED  [ 10%]
tests/test_magika_python_cli.py::test_magika_cli_with_directories FAILED [ [12](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:13)%]
tests/test_magika_python_cli.py::test_magika_cli_with_symlinks FAILED    [ [15](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:16)%]
tests/test_magika_python_cli.py::test_magika_cli_with_files_with_permission_errors FAILED [ [17](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:18)%]
tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files FAILED [ [20](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:21)%]
tests/test_magika_python_cli.py::test_magika_cli_with_mitra_test_files FAILED [ [23](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:24)%]
tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_json_output FAILED [ [25](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:26)%]
tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_jsonl_output FAILED [ [28](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:29)%]
tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_probability FAILED [ [30](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:31)%]
tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_mime_output FAILED [ [33](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:34)%]
tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_label_output FAILED [ [35](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:36)%]
tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_compatibility_mode FAILED [ [38](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:39)%]
tests/test_magika_python_cli.py::test_magika_cli_output_with_low_confidence_prediction FAILED [ [41](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:42)%]
tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_different_prediction_modes FAILED [ [43](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:44)%]
tests/test_magika_python_cli.py::test_magika_cli_with_python_and_not_python_files FAILED [ [46](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:47)%]
tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_custom_batch_sizes FAILED [ [48](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:49)%]
tests/test_magika_python_cli.py::test_magika_cli_with_multiple_copies_of_the_same_file FAILED [ [51](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:52)%]
tests/test_magika_python_cli.py::test_magika_cli_with_many_files FAILED  [ [53](https://github.com/gaby/magika/actions/runs/7982383780/job/21796809351#step:6:54)%]
tests/test_magika_python_cli.py::test_magika_cli_with_bad_input FAILED   [ 56%]
tests/test_magika_python_cli.py::test_magika_cli_with_reading_from_stdin PASSED [ 58%]
tests/test_magika_python_cli.py::test_magika_cli_with_colors FAILED      [ 61%]
tests/test_magika_python_cli.py::test_magika_cli_with_no_colors FAILED   [ 64%]
tests/test_magika_python_cli.py::test_magika_cli_generate_report FAILED  [ 66%]
tests/test_magika_python_cli.py::test_magika_cli_output_version FAILED   [ 69%]
tests/test_magika_python_cli.py::test_magika_cli_help FAILED             [ 71%]
tests/test_magika_python_cli.py::test_magika_cli_list_content_types FAILED [ 74%]
tests/test_magika_python_module.py::test_magika_module_one_basic_test PASSED [ 76%]
tests/test_magika_python_module.py::test_magika_module_with_default_model PASSED [ 79%]
tests/test_magika_python_module.py::test_magika_module_with_basic_tests_by_paths PASSED [ 82%]
tests/test_magika_python_module.py::test_magika_module_with_basic_tests_by_path PASSED [ 84%]
tests/test_magika_python_module.py::test_magika_module_with_basic_tests_by_bytes PASSED [ 87%]
tests/test_magika_python_module.py::test_magika_module_with_empty_content PASSED [ 89%]
tests/test_magika_python_module.py::test_magika_module_with_short_content PASSED [ 92%]
tests/test_magika_python_module.py::test_magika_module_with_different_prediction_modes PASSED [ 94%]
tests/test_magika_python_module.py::test_extract_features_with_ascii PASSED [ 97%]
tests/test_magika_python_module.py::test_extract_features_with_spaces PASSED [100%]

C:\hostedtoolcache\windows\Python\3.11.8\x64\Lib\subprocess.py:1538: FileNotFoundError
=========================== short test summary info ===========================
FAILED tests/test_magika_python_cli.py::test_magika_python_cli_with_one_test_file - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_python_cli_with_very_small_test_files - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_small_test_files - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_empty_file - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_directories - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_symlinks - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_files_with_permission_errors - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_mitra_test_files - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_json_output - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_jsonl_output - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_probability - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_mime_output - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_label_output - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_compatibility_mode - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_output_with_low_confidence_prediction - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_different_prediction_modes - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_python_and_not_python_files - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_basic_test_files_and_custom_batch_sizes - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_multiple_copies_of_the_same_file - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_many_files - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_bad_input - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_colors - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_with_no_colors - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_generate_report - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_output_version - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_help - FileNotFoundError: [WinError 2] The system cannot find the file specified
FAILED tests/test_magika_python_cli.py::test_magika_cli_list_content_types - FileNotFoundError: [WinError 2] The system cannot find the file specified
================= 28 failed, 11 passed, 2 deselected in 7.31s =================

gaby avatar Feb 21 '24 01:02 gaby

I can't get the unit-test to run on Windows, tried 20 different ways and using a different poetry action. All the tests that are using subprocess fail with Windows.

tests/test_magika_python_module.py::test_magika_module_with_different_prediction_modes PASSED [100%]
================================== FAILURES ===================================
__________________ test_magika_python_cli_with_one_test_file __________________
    @pytest.mark.smoketest
    def test_magika_python_cli_with_one_test_file() -> None:
        test_file_path = utils.get_basic_test_files_paths()[0]
    
>       stdout, stderr = run_magika_python_cli([test_file_path])
tests\test_magika_python_cli.py:33: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests\utils_magika_python_client.py:81: in run_magika_python_cli
    p = subprocess.run(cmd, capture_output=True, text=True, check=False)
C:\hostedtoolcache\windows\Python\3.12.2\x64\Lib\subprocess.py:548: in run
    with Popen(*popenargs, **kwargs) as process:
C:\hostedtoolcache\windows\Python\3.12.2\x64\Lib\subprocess.py:1026: in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <Popen: returncode: None args: ['magika', 'D:\\a\\magika\\magika\\tests_data...>
args = 'magika D:\\a\\magika\\magika\\tests_data\\basic\\code.asm --no-colors'
executable = None, preexec_fn = None, close_fds = False, pass_fds = ()
cwd = None, env = None
startupinfo = <subprocess.STARTUPINFO object at 0x00000273C6AF5B80>
creationflags = 0, shell = False, p2cread = Handle(188), p2cwrite = -1
c2pread = 12, c2pwrite = Handle(420), errread = 13, errwrite = Handle([60](https://github.com/gaby/magika/actions/runs/8040664470/job/21959022545#step:7:61)4)
unused_restore_signals = True, unused_gid = None, unused_gids = None
unused_uid = None, unused_umask = -1, unused_start_new_session = False
unused_process_group = -1
    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       unused_restore_signals,
                       unused_gid, unused_gids, unused_uid,
                       unused_umask,
                       unused_start_new_session, unused_process_group):
        """Execute program (MS Windows version)"""
    
        assert not pass_fds, "pass_fds not supported on Windows."
    
        if isinstance(args, str):
            pass
        elif isinstance(args, bytes):
            if shell:
                raise TypeError('bytes args is not allowed on Windows')
            args = list2cmdline([args])
        elif isinstance(args, os.PathLike):
            if shell:
                raise TypeError('path-like args is not allowed when '
                                'shell is true')
            args = list2cmdline([args])
        else:
            args = list2cmdline(args)
    
        if executable is not None:
            executable = os.fsdecode(executable)
    
        # Process startup details
        if startupinfo is None:
            startupinfo = STARTUPINFO()
        else:
            # bpo-34044: Copy STARTUPINFO since it is modified above,
            # so the caller can reuse it multiple times.
            startupinfo = startupinfo.copy()
    
        use_std_handles = -1 not in (p2cread, c2pwrite, errwrite)
        if use_std_handles:
            startupinfo.dwFlags |= _winapi.STARTF_USESTDHANDLES
            startupinfo.hStdInput = p2cread
            startupinfo.hStdOutput = c2pwrite
            startupinfo.hStdError = errwrite
    
        attribute_list = startupinfo.lpAttributeList
        have_handle_list = bool(attribute_list and
                                "handle_list" in attribute_list and
                                attribute_list["handle_list"])
    
        # If we were given an handle_list or need to create one
        if have_handle_list or (use_std_handles and close_fds):
            if attribute_list is None:
                attribute_list = startupinfo.lpAttributeList = {}
            handle_list = attribute_list["handle_list"] = \
                list(attribute_list.get("handle_list", []))
    
            if use_std_handles:
                handle_list += [int(p2cread), int(c2pwrite), int(errwrite)]
    
            handle_list[:] = self._filter_handle_list(handle_list)
    
            if handle_list:
                if not close_fds:
                    warnings.warn("startupinfo.lpAttributeList['handle_list'] "
                                  "overriding close_fds", RuntimeWarning)
    
                # When using the handle_list we always request to inherit
                # handles but the only handles that will be inherited are
                # the ones in the handle_list
                close_fds = False
    
        if shell:
            startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
            startupinfo.wShowWindow = _winapi.SW_HIDE
            if not executable:
                # gh-101283: without a fully-qualified path, before Windows
                # checks the system directories, it first looks in the
                # application directory, and also the current directory if
                # NeedCurrentDirectoryForExePathW(ExeName) is true, so try
                # to avoid executing unqualified "cmd.exe".
                comspec = os.environ.get('ComSpec')
                if not comspec:
                    system_root = os.environ.get('SystemRoot', '')
                    comspec = os.path.join(system_root, 'System32', 'cmd.exe')
                    if not os.path.isabs(comspec):
                        raise FileNotFoundError('shell not found: neither %ComSpec% nor %SystemRoot% is set')
                if os.path.isabs(comspec):
                    executable = comspec
            else:
                comspec = executable
    
            args = '{} /c "{}"'.format (comspec, args)
    
        if cwd is not None:
            cwd = os.fsdecode(cwd)
    
        sys.audit("subprocess.Popen", executable, args, cwd, env)
    
        # Start the process
        try:
>           hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                                     # no special security
                                     None, None,
                                     int(not close_fds),
                                     creationflags,
                                     env,
                                     cwd,
                                     startupinfo)
E                                    FileNotFoundError: [WinError 2] The system cannot find the file specified
C:\hostedtoolcache\windows\Python\3.12.2\x[64](https://github.com/gaby/magika/actions/runs/8040664470/job/21959022545#step:7:65)\Lib\subprocess.py:1538: FileNotFoundError
______________ test_magika_python_cli_with_very_small_test_files ______________
    def test_magika_python_cli_with_very_small_test_files() -> None:
        """Magika does not use the DL model for very small files. This test covers
        these scenarios.
        """
    
        with tempfile.TemporaryDirectory() as td:
            text_test_path = Path(td) / "small.txt"
            text_test_path.write_text("small test")
>           stdout, stderr = run_magika_python_cli([text_test_path], label_output=True)
tests\test_magika_python_cli.py:89: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests\utils_magika_python_client.py:[81](https://github.com/gaby/magika/actions/runs/8040664470/job/21959022545#step:7:82): in run_magika_python_cli
    p = subprocess.run(cmd, capture_output=True, text=True, check=False)
C:\hostedtoolcache\windows\Python\3.12.2\x64\Lib\subprocess.py:548: in run
    with Popen(*popenargs, **kwargs) as process:
C:\hostedtoolcache\windows\Python\3.12.2\x64\Lib\subprocess.py:[102](https://github.com/gaby/magika/actions/runs/8040664470/job/21959022545#step:7:103)6: in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

After installing and running magika --version and magika --help works, but pytest can't find the cli to run the test

gaby avatar Feb 25 '24 20:02 gaby

@reyammer Found something interesting, the workflow in https://github.com/google/magika/pull/254 confirms that magika does indeed work on Windows. The issue is related pytest/unit-tests.

gaby avatar Feb 26 '24 03:02 gaby