lxml
lxml copied to clipboard
amauryfa
Building on PyPy 2.6 (official docker image) breaks with CompileError
Building on PyPy 2.6 (with cffi 1.1) breaks with ValidationError whereas it works perfectly on PyPy 2.5. Tests were made with official PyPy docker images (tags '2-2.6' vs '2-2.5').
PyPy 2.6:
npiganeau@ndp-host:~$ docker run -t -i --name pypy pypy:2-2.6 bash
root@8557c3209fe5:/# pip install -e git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi
Obtaining lxml-cffi from git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi
Cloning git://github.com/amauryfa/lxml.git (to cffi) to /src/lxml-cffi
Complete output from command python setup.py egg_info:
src/lxml-cffi/includes/__pycache__/_cffi__gf1d3c271xb1c86544.c: In function ‘_cffi_check_struct__xmlOutputBuffer’:
src/lxml-cffi/includes/__pycache__/_cffi__gf1d3c271xb1c86544.c:1624:24: warning: initialization from incompatible pointer type
{ xmlBuffer * *tmp = &p->buffer; (void)tmp; }
^
src/lxml-cffi/includes/__pycache__/_cffi__gf1d3c271xb1c86544.c:1625:24: warning: initialization from incompatible pointer type
{ xmlBuffer * *tmp = &p->conv; (void)tmp; }
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c: In function ‘_cffi_e__xmlRelaxNGValidErr’:
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7845:8: error: ‘XML_RELAXNG_OK’ undeclared (first use in this function)
if ((XML_RELAXNG_OK) > 0 || (long)(XML_RELAXNG_OK) != 0L) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7845:8: note: each undeclared identifier is reported only once for each function it appears in
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7855:8: error: ‘XML_RELAXNG_ERR_MEMORY’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_MEMORY) <= 0 || (unsigned long)(XML_RELAXNG_ERR_MEMORY) != 1UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7865:8: error: ‘XML_RELAXNG_ERR_TYPE’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_TYPE) <= 0 || (unsigned long)(XML_RELAXNG_ERR_TYPE) != 2UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7875:8: error: ‘XML_RELAXNG_ERR_TYPEVAL’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_TYPEVAL) <= 0 || (unsigned long)(XML_RELAXNG_ERR_TYPEVAL) != 3UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7885:8: error: ‘XML_RELAXNG_ERR_DUPID’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_DUPID) <= 0 || (unsigned long)(XML_RELAXNG_ERR_DUPID) != 4UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7895:8: error: ‘XML_RELAXNG_ERR_TYPECMP’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_TYPECMP) <= 0 || (unsigned long)(XML_RELAXNG_ERR_TYPECMP) != 5UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7905:8: error: ‘XML_RELAXNG_ERR_NOSTATE’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_NOSTATE) <= 0 || (unsigned long)(XML_RELAXNG_ERR_NOSTATE) != 6UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7915:8: error: ‘XML_RELAXNG_ERR_NODEFINE’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_NODEFINE) <= 0 || (unsigned long)(XML_RELAXNG_ERR_NODEFINE) != 7UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7925:8: error: ‘XML_RELAXNG_ERR_LISTEXTRA’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_LISTEXTRA) <= 0 || (unsigned long)(XML_RELAXNG_ERR_LISTEXTRA) != 8UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7935:8: error: ‘XML_RELAXNG_ERR_LISTEMPTY’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_LISTEMPTY) <= 0 || (unsigned long)(XML_RELAXNG_ERR_LISTEMPTY) != 9UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7945:8: error: ‘XML_RELAXNG_ERR_INTERNODATA’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_INTERNODATA) <= 0 || (unsigned long)(XML_RELAXNG_ERR_INTERNODATA) != 10UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7955:8: error: ‘XML_RELAXNG_ERR_INTERSEQ’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_INTERSEQ) <= 0 || (unsigned long)(XML_RELAXNG_ERR_INTERSEQ) != 11UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7965:8: error: ‘XML_RELAXNG_ERR_INTEREXTRA’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_INTEREXTRA) <= 0 || (unsigned long)(XML_RELAXNG_ERR_INTEREXTRA) != 12UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7975:8: error: ‘XML_RELAXNG_ERR_ELEMNAME’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_ELEMNAME) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ELEMNAME) != 13UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7985:8: error: ‘XML_RELAXNG_ERR_ATTRNAME’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_ATTRNAME) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ATTRNAME) != 14UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:7995:8: error: ‘XML_RELAXNG_ERR_ELEMNONS’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_ELEMNONS) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ELEMNONS) != 15UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8005:8: error: ‘XML_RELAXNG_ERR_ATTRNONS’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_ATTRNONS) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ATTRNONS) != 16UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8015:8: error: ‘XML_RELAXNG_ERR_ELEMWRONGNS’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_ELEMWRONGNS) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ELEMWRONGNS) != 17UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8025:8: error: ‘XML_RELAXNG_ERR_ATTRWRONGNS’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_ATTRWRONGNS) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ATTRWRONGNS) != 18UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8035:8: error: ‘XML_RELAXNG_ERR_ELEMEXTRANS’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_ELEMEXTRANS) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ELEMEXTRANS) != 19UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8045:8: error: ‘XML_RELAXNG_ERR_ATTREXTRANS’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_ATTREXTRANS) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ATTREXTRANS) != 20UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8055:8: error: ‘XML_RELAXNG_ERR_ELEMNOTEMPTY’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_ELEMNOTEMPTY) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ELEMNOTEMPTY) != 21UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8065:8: error: ‘XML_RELAXNG_ERR_NOELEM’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_NOELEM) <= 0 || (unsigned long)(XML_RELAXNG_ERR_NOELEM) != 22UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8075:8: error: ‘XML_RELAXNG_ERR_NOTELEM’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_NOTELEM) <= 0 || (unsigned long)(XML_RELAXNG_ERR_NOTELEM) != 23UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8085:8: error: ‘XML_RELAXNG_ERR_ATTRVALID’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_ATTRVALID) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ATTRVALID) != 24UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8095:8: error: ‘XML_RELAXNG_ERR_CONTENTVALID’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_CONTENTVALID) <= 0 || (unsigned long)(XML_RELAXNG_ERR_CONTENTVALID) != 25UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8105:8: error: ‘XML_RELAXNG_ERR_EXTRACONTENT’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_EXTRACONTENT) <= 0 || (unsigned long)(XML_RELAXNG_ERR_EXTRACONTENT) != 26UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8115:8: error: ‘XML_RELAXNG_ERR_INVALIDATTR’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_INVALIDATTR) <= 0 || (unsigned long)(XML_RELAXNG_ERR_INVALIDATTR) != 27UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8125:8: error: ‘XML_RELAXNG_ERR_DATAELEM’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_DATAELEM) <= 0 || (unsigned long)(XML_RELAXNG_ERR_DATAELEM) != 28UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8135:8: error: ‘XML_RELAXNG_ERR_VALELEM’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_VALELEM) <= 0 || (unsigned long)(XML_RELAXNG_ERR_VALELEM) != 29UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8145:8: error: ‘XML_RELAXNG_ERR_LISTELEM’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_LISTELEM) <= 0 || (unsigned long)(XML_RELAXNG_ERR_LISTELEM) != 30UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8155:8: error: ‘XML_RELAXNG_ERR_DATATYPE’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_DATATYPE) <= 0 || (unsigned long)(XML_RELAXNG_ERR_DATATYPE) != 31UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8165:8: error: ‘XML_RELAXNG_ERR_VALUE’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_VALUE) <= 0 || (unsigned long)(XML_RELAXNG_ERR_VALUE) != 32UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8175:8: error: ‘XML_RELAXNG_ERR_LIST’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_LIST) <= 0 || (unsigned long)(XML_RELAXNG_ERR_LIST) != 33UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8185:8: error: ‘XML_RELAXNG_ERR_NOGRAMMAR’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_NOGRAMMAR) <= 0 || (unsigned long)(XML_RELAXNG_ERR_NOGRAMMAR) != 34UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8195:8: error: ‘XML_RELAXNG_ERR_EXTRADATA’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_EXTRADATA) <= 0 || (unsigned long)(XML_RELAXNG_ERR_EXTRADATA) != 35UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8205:8: error: ‘XML_RELAXNG_ERR_LACKDATA’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_LACKDATA) <= 0 || (unsigned long)(XML_RELAXNG_ERR_LACKDATA) != 36UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8215:8: error: ‘XML_RELAXNG_ERR_INTERNAL’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_INTERNAL) <= 0 || (unsigned long)(XML_RELAXNG_ERR_INTERNAL) != 37UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8225:8: error: ‘XML_RELAXNG_ERR_ELEMWRONG’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_ELEMWRONG) <= 0 || (unsigned long)(XML_RELAXNG_ERR_ELEMWRONG) != 38UL) {
^
src/lxml-cffi/__pycache__/_cffi__g65f9f103xa680a747.c:8235:8: error: ‘XML_RELAXNG_ERR_TEXTWRONG’ undeclared (first use in this function)
if ((XML_RELAXNG_ERR_TEXTWRONG) <= 0 || (unsigned long)(XML_RELAXNG_ERR_TEXTWRONG) != 39UL) {
^
Building lxml version 3.4.0.
Building without Cython.
Using build configuration of libxslt 1.1.28
Traceback (most recent call last):
File "<builtin>/app_main.py", line 75, in run_toplevel
File "<builtin>/app_main.py", line 601, in run_it
File "<string>", line 20, in <module>
File "/src/lxml-cffi/setup.py", line 232, in <module>
**setup_extra_options()
File "/src/lxml-cffi/setup.py", line 145, in setup_extra_options
STATIC_CFLAGS, STATIC_BINARIES)
File "setupinfo.py", line 167, in ext_modules
import lxml.etree
File "src/lxml/../lxml-cffi/etree.py", line 26, in <module>
from .xmlerror import _initThreadLogging, clear_error_log
File "src/lxml/../lxml-cffi/xmlerror.py", line 150, in <module>
libraries=['xml2'])
File "/usr/local/lib_pypy/cffi/api.py", line 373, in verify
lib = self.verifier.load_library()
File "/usr/local/lib_pypy/cffi/verifier.py", line 96, in load_library
self._compile_module()
File "/usr/local/lib_pypy/cffi/verifier.py", line 192, in _compile_module
outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
File "/usr/local/lib_pypy/cffi/ffiplatform.py", line 38, in compile
outputfilename = _build(tmpdir, ext)
File "/usr/local/lib_pypy/cffi/ffiplatform.py", line 65, in _build
raise VerificationError('%s: %s' % (e.__class__.__name__, e))
VerificationError: CompileError: command 'cc' failed with exit status 1
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /src/lxml-cffi
PyPy 2.5
npiganeau@ndp-host:~$ docker run -t -i --name pypy pypy:2-2.5 bash
root@6b84fa93cfd9:/# pip install -e git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi
You are using pip version 7.0.1, however version 7.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Obtaining lxml-cffi from git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi
Cloning git://github.com/amauryfa/lxml.git (to cffi) to /src/lxml-cffi
Installing collected packages: lxml-cffi
Running setup.py develop for lxml-cffi
Successfully installed lxml-cffi
I believe these bindings are outdated and cannot be used with today's PyPy. After much fiddling with the sources I was able to build it on OSX 10.9.5 (Maverics) / macports + PyPy 2.6. However it was super unstable, spat out RuntimeError when querying an elem for its children (in random places). Totally unusable IMO. So I deleted it all together.
The error messages seem to indicate a different version of libxml, PyPy's version is not relevant here. Can you check which version of libxml you are using?
@amauryfa Actually it is the same version of libxml in both docker instances: 2.9.1.
root@6689ce5271c5:/# pypy --version
Python 2.7.9 (9c4588d731b7, Mar 23 2015, 16:30:30)
[PyPy 2.5.1 with GCC 4.6.3]
root@6689ce5271c5:/# ldconfig -v|grep xml
libxml2.so.2 -> libxml2.so.2.9.1
root@3b5593dee3af:/# pypy --version
Python 2.7.9 (295ee98b6928, May 31 2015, 07:29:04)
[PyPy 2.6.0 with GCC 4.8.2]
root@3b5593dee3af:/# ldconfig -v |grep xml
libxml2.so.2 -> libxml2.so.2.9.1
And while we're at it, in my tests I noticed it took the cffi+pypy lxml ~3-5 times more time than cpython lxml to parse the same HTML page. I wasn't expecting that at all, thought cffi would be faster. Have you seen anything like it?
2015-06-21 22:41 GMT+02:00 Aleksandr Glyzov [email protected]:
And while we're at it, in my tests I noticed it took the cffi+pypy lxml ~3-5 times more time than cpython lxml to parse the same HTML page. I wasn't expecting that at all, thought cffi would be faster. Have you seen anything like it?
I remember doing some benchmarks, and the results were in the same range as CPython (some were 40% faster, others 40% slower...) But it's very possible that some paths are still much slower.
Also, be sure to warm up pypy's JIT by running 1000 loops of your function before doing any timings.
Amaury Forgeot d'Arc
@nichochar and @aldarund guys, try this fork: https://github.com/aglyzov/lxml.git I have added explicit libxml/relaxng.h includes everywhere and it started to build ok.