yara-python
yara-python copied to clipboard
VirusTotal
Module "magic" Not Working On macOS
When installing yara-python using pip on macOS, the following exception is raised:
$ python test.py
Traceback (most recent call last):
File "test.py", line 3, in <module>
rules = yara.compile(filepath='test.yar')
yara.SyntaxError: test.yar(6): invalid field name "mime_type"
This is the YARA test file:
import "magic"
rule test
{
condition:
magic.mime_type() == "application/vnd.ms-excel"
}
This is the Python testing script:
import yara
rules = yara.compile(filepath='test.yar')
matches = rules.match('test.txt')
print(matches)
If yara-python is installed from source, the following exception is raised:
$ python test.py
Traceback (most recent call last):
File "test.py", line 1, in <module>
import yara
ImportError: dlopen(/Users/username/Desktop/test/venv/lib/python3.8/site-packages/yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg/yara.cpython-38-darwin.so, 2): Symbol not found: _yr_compiler_
add_file
Referenced from: /Users/username/Desktop/test/venv/lib/python3.8/site-packages/yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg/yara.cpython-38-darwin.so
Expected in: flat namespace
in /Users/username/Desktop/test/venv/lib/python3.8/site-packages/yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg/yara.cpython-38-darwin.so
Here are the steps I used to install from pip:
$ python3 -m venv venv
$ source venv/bin/activate
(venv) $ pip install -U pip setuptools
Collecting pip
Using cached pip-20.2-py2.py3-none-any.whl (1.5 MB)
Collecting setuptools
Using cached setuptools-49.2.1-py3-none-any.whl (789 kB)
Installing collected packages: pip, setuptools
Attempting uninstall: pip
Found existing installation: pip 20.1.1
Uninstalling pip-20.1.1:
Successfully uninstalled pip-20.1.1
Attempting uninstall: setuptools
Found existing installation: setuptools 47.1.0
Uninstalling setuptools-47.1.0:
Successfully uninstalled setuptools-47.1.0
Successfully installed pip-20.2 setuptools-49.2.1
(venv) $ pip install yara-python
Collecting yara-python
Using cached yara-python-4.0.2.tar.gz (405 kB)
Using legacy 'setup.py install' for yara-python, since package 'wheel' is not installed.
Installing collected packages: yara-python
Running setup.py install for yara-python ... done
Successfully installed yara-python-4.0.2
Here are the steps I used to install from source:
$ python3 -m venv venv
$ source venv/bin/activate
(venv) $ pip install -U pip setuptools
Collecting pip
Using cached pip-20.2-py2.py3-none-any.whl (1.5 MB)
Collecting setuptools
Using cached setuptools-49.2.1-py3-none-any.whl (789 kB)
Installing collected packages: pip, setuptools
Attempting uninstall: pip
Found existing installation: pip 20.1.1
Uninstalling pip-20.1.1:
Successfully uninstalled pip-20.1.1
Attempting uninstall: setuptools
Found existing installation: setuptools 47.1.0
Uninstalling setuptools-47.1.0:
Successfully uninstalled setuptools-47.1.0
Successfully installed pip-20.2 setuptools-49.2.1
(venv) $ git clone [email protected]:VirusTotal/yara-python.git
Cloning into 'yara-python'...
remote: Enumerating objects: 24, done.
remote: Counting objects: 100% (24/24), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 969 (delta 10), reused 18 (delta 6), pack-reused 945
Receiving objects: 100% (969/969), 231.01 KiB | 17.77 MiB/s, done.
Resolving deltas: 100% (570/570), done.
(venv) $ python setup.py build
/Users/username/Desktop/test/venv/lib/python3.8/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit
undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
warnings.warn(
running build
running build_ext
building 'yara' extension
creating build
creating build/temp.macosx-10.15-x86_64-3.8
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -I/Library/Dev
eloper/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/op
t/[email protected]/include -D_GNU_SOURCE=1 -DUSE_MACH_PROC=1 -DHAVE_MEMMEM=1 -Iyara/libyara/include -Iyara/libyara/ -I. -I/usr/local/opt/openssl/include -I/opt/local/include -I/usr/local/include -
I/usr/local/include -I/usr/local/opt/[email protected]/include -I/usr/local/opt/sqlite/include -I/Users/username/Desktop/test/venv/include -I/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.fram
ework/Versions/3.8/include/python3.8 -c yara-python.c -o build/temp.macosx-10.15-x86_64-3.8/yara-python.o -std=c99
yara-python.c:442:27: warning: comparison of integers of different signs: 'int64_t' (aka 'long long') and 'unsigned long long' [-Wsign-compare]
if (object->value.i != YR_UNDEFINED)
~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~
yara-python.c:993:16: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
identifier = PY_STRING_TO_C(key);
^ ~~~~~~~~~~~~~~~~~~~
yara-python.c:1022:13: warning: initializing 'char *' with an expression of type 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
char* str = PY_STRING_TO_C(value);
^ ~~~~~~~~~~~~~~~~~~~~~
yara-python.c:1063:16: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
identifier = PY_STRING_TO_C(key);
^ ~~~~~~~~~~~~~~~~~~~
yara-python.c:1092:13: warning: initializing 'char *' with an expression of type 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
char* str = PY_STRING_TO_C(value);
^ ~~~~~~~~~~~~~~~~~~~~~
yara-python.c:2089:18: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
source = PY_STRING_TO_C(value);
^ ~~~~~~~~~~~~~~~~~~~~~
yara-python.c:2090:14: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
ns = PY_STRING_TO_C(key);
^ ~~~~~~~~~~~~~~~~~~~
yara-python.c:2122:20: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
filepath = PY_STRING_TO_C(value);
^ ~~~~~~~~~~~~~~~~~~~~~
yara-python.c:2123:14: warning: assigning to 'char *' from 'const char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
ns = PY_STRING_TO_C(key);
^ ~~~~~~~~~~~~~~~~~~~
9 warnings generated.
creating build/lib.macosx-10.15-x86_64-3.8
clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk -L/usr/local/opt/[email protected]/lib -I/usr/local/opt/[email protected]/include build/temp.mac
osx-10.15-x86_64-3.8/yara-python.o -L/opt/local/lib -L/usr/local/lib -L/usr/local/lib -L/usr/local/opt/[email protected]/lib -L/usr/local/opt/sqlite/lib -o build/lib.macosx-10.15-x86_64-3.8/yara.cp
ython-38-darwin.so
ld: warning: directory not found for option '-L/opt/local/lib'
(venv) $ python setup.py install
/Users/username/Desktop/test/venv/lib/python3.8/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit
undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
warnings.warn(
running install
running bdist_egg
running egg_info
creating yara_python.egg-info
writing yara_python.egg-info/PKG-INFO
writing dependency_links to yara_python.egg-info/dependency_links.txt
writing top-level names to yara_python.egg-info/top_level.txt
writing manifest file 'yara_python.egg-info/SOURCES.txt'
reading manifest file 'yara_python.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'yara/libyara/modules/module_list'
warning: no files found matching '*.c' under directory 'yara'
warning: no files found matching '*.h' under directory 'yara'
writing manifest file 'yara_python.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.15-x86_64/egg
running install_lib
running build_ext
creating build/bdist.macosx-10.15-x86_64
creating build/bdist.macosx-10.15-x86_64/egg
copying build/lib.macosx-10.15-x86_64-3.8/yara.cpython-38-darwin.so -> build/bdist.macosx-10.15-x86_64/egg
creating stub loader for yara.cpython-38-darwin.so
byte-compiling build/bdist.macosx-10.15-x86_64/egg/yara.py to yara.cpython-38.pyc
creating build/bdist.macosx-10.15-x86_64/egg/EGG-INFO
copying yara_python.egg-info/PKG-INFO -> build/bdist.macosx-10.15-x86_64/egg/EGG-INFO
copying yara_python.egg-info/SOURCES.txt -> build/bdist.macosx-10.15-x86_64/egg/EGG-INFO
copying yara_python.egg-info/dependency_links.txt -> build/bdist.macosx-10.15-x86_64/egg/EGG-INFO
copying yara_python.egg-info/not-zip-safe -> build/bdist.macosx-10.15-x86_64/egg/EGG-INFO
copying yara_python.egg-info/top_level.txt -> build/bdist.macosx-10.15-x86_64/egg/EGG-INFO
writing build/bdist.macosx-10.15-x86_64/egg/EGG-INFO/native_libs.txt
creating dist
creating 'dist/yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg' and adding 'build/bdist.macosx-10.15-x86_64/egg' to it
removing 'build/bdist.macosx-10.15-x86_64/egg' (and everything under it)
Processing yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg
creating /Users/username/Desktop/test/venv/lib/python3.8/site-packages/yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg
Extracting yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg to /Users/username/Desktop/test/venv/lib/python3.8/site-packages
Adding yara-python 4.0.2 to easy-install.pth file
Installed /Users/username/Desktop/test/venv/lib/python3.8/site-packages/yara_python-4.0.2-py3.8-macosx-10.15-x86_64.egg
Processing dependencies for yara-python==4.0.2
Finished processing dependencies for yara-python==4.0.2
Everything works if the build is dynamically linked to the installed libyara: python setup.py build --dynamic-linking
When installing yara-python using pip on macOS, the following exception is raised:
This is expected. The magic module is not on by default, and you did not tell pip to build with that option.
If yara-python is installed from source, the following exception is raised:
This part is a local problem and is likely because you built yara-python from src without having the yara submodule, so it used whatever old version of libyara it found laying around your system which does not include the necessary symbols (yr_compiler_add_file). When you clone yara-python clone it with the --recursive flag.
The above problem is likely the same occurrence as #8
Just to make it clear for other people who encounter this problem, yara-python needs to be installed from source rather than using pip.
It would be nice to craft a method where a pip installed yara-python just worked without building from source, but until that time:
git clone --recursive [email protected]:VirusTotal/yara-python.git
python setup.py build --dynamic-linking
python setup.py install