mdanalysis GSD tests are failing on big-endian architectures

Expected behaviour

All tests pass.

Actual behaviour

==================================== ERRORS ====================================
_____________________ ERROR at setup of test_gsd_data_step _____________________
[gw1] linux2 -- Python 2.7.14 /usr/bin/python2
    @pytest.fixture
    def GSD_U():
>       return mda.Universe(GSD)
coordinates/test_gsd.py:34: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/core/universe.py:261: in __init__
    self._topology = p.parse(**kwargs)
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/topology/GSDParser.py:107: in parse
    with gsd.hoomd.open(self.filename,mode='rb') as t :
/usr/lib64/python2.7/site-packages/gsd/hoomd.py:799: in open
    schema_version=[1,2]);
fl.pyx:163: in gsd.fl.open
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: Not a GSD file: /builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd
fl.pyx:279: RuntimeError
_____________________ ERROR at setup of test_gsd_positions _____________________
[gw3] linux2 -- Python 2.7.14 /usr/bin/python2
    @pytest.fixture
    def GSD_U():
>       return mda.Universe(GSD)
coordinates/test_gsd.py:34: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/core/universe.py:261: in __init__
    self._topology = p.parse(**kwargs)
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/topology/GSDParser.py:107: in parse
    with gsd.hoomd.open(self.filename,mode='rb') as t :
/usr/lib64/python2.7/site-packages/gsd/hoomd.py:799: in open
    schema_version=[1,2]);
fl.pyx:163: in gsd.fl.open
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: Not a GSD file: /builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd
fl.pyx:279: RuntimeError
____________________ ERROR at setup of test_gsd_dimensions _____________________
[gw0] linux2 -- Python 2.7.14 /usr/bin/python2
    @pytest.fixture
    def GSD_U():
>       return mda.Universe(GSD)
coordinates/test_gsd.py:34: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/core/universe.py:261: in __init__
    self._topology = p.parse(**kwargs)
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/topology/GSDParser.py:107: in parse
    with gsd.hoomd.open(self.filename,mode='rb') as t :
/usr/lib64/python2.7/site-packages/gsd/hoomd.py:799: in open
    schema_version=[1,2]);
fl.pyx:163: in gsd.fl.open
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: Not a GSD file: /builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd
fl.pyx:279: RuntimeError
_____________________ ERROR at setup of test_gsd_n_frames ______________________
[gw2] linux2 -- Python 2.7.14 /usr/bin/python2
    @pytest.fixture
    def GSD_U():
>       return mda.Universe(GSD)
coordinates/test_gsd.py:34: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/core/universe.py:261: in __init__
    self._topology = p.parse(**kwargs)
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/topology/GSDParser.py:107: in parse
    with gsd.hoomd.open(self.filename,mode='rb') as t :
/usr/lib64/python2.7/site-packages/gsd/hoomd.py:799: in open
    schema_version=[1,2]);
fl.pyx:163: in gsd.fl.open
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: Not a GSD file: /builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd
fl.pyx:279: RuntimeError
__________ ERROR at setup of TestGSDParser.test_mandatory_attributes ___________
[gw0] linux2 -- Python 2.7.14 /usr/bin/python2
self = <MDAnalysisTests.topology.test_gsd.TestGSDParser object at 0x7fff987e9c50>
filename = '/builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd'
    @pytest.fixture()
    def top(self, filename):
        with self.parser(filename) as p:
>           yield p.parse()
topology/base.py:46: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/topology/GSDParser.py:107: in parse
    with gsd.hoomd.open(self.filename,mode='rb') as t :
/usr/lib64/python2.7/site-packages/gsd/hoomd.py:799: in open
    schema_version=[1,2]);
fl.pyx:163: in gsd.fl.open
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: Not a GSD file: /builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd
fl.pyx:279: RuntimeError
___________ ERROR at setup of TestGSDParser.test_expected_attributes ___________
[gw0] linux2 -- Python 2.7.14 /usr/bin/python2
self = <MDAnalysisTests.topology.test_gsd.TestGSDParser object at 0x7fff9e0c8e50>
filename = '/builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd'
    @pytest.fixture()
    def top(self, filename):
        with self.parser(filename) as p:
>           yield p.parse()
topology/base.py:46: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/topology/GSDParser.py:107: in parse
    with gsd.hoomd.open(self.filename,mode='rb') as t :
/usr/lib64/python2.7/site-packages/gsd/hoomd.py:799: in open
    schema_version=[1,2]);
fl.pyx:163: in gsd.fl.open
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: Not a GSD file: /builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd
fl.pyx:279: RuntimeError
___________ ERROR at setup of TestGSDParser.test_guessed_attributes ____________
[gw0] linux2 -- Python 2.7.14 /usr/bin/python2
self = <MDAnalysisTests.topology.test_gsd.TestGSDParser object at 0x7fffa1cef890>
filename = '/builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd'
    @pytest.fixture()
    def top(self, filename):
        with self.parser(filename) as p:
>           yield p.parse()
topology/base.py:46: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/topology/GSDParser.py:107: in parse
    with gsd.hoomd.open(self.filename,mode='rb') as t :
/usr/lib64/python2.7/site-packages/gsd/hoomd.py:799: in open
    schema_version=[1,2]);
fl.pyx:163: in gsd.fl.open
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: Not a GSD file: /builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd
fl.pyx:279: RuntimeError
__________________ ERROR at setup of TestGSDParser.test_size ___________________
[gw0] linux2 -- Python 2.7.14 /usr/bin/python2
self = <MDAnalysisTests.topology.test_gsd.TestGSDParser object at 0x7fffa17e2d90>
filename = '/builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd'
    @pytest.fixture()
    def top(self, filename):
        with self.parser(filename) as p:
>           yield p.parse()
topology/base.py:46: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/topology/GSDParser.py:107: in parse
    with gsd.hoomd.open(self.filename,mode='rb') as t :
/usr/lib64/python2.7/site-packages/gsd/hoomd.py:799: in open
    schema_version=[1,2]);
fl.pyx:163: in gsd.fl.open
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: Not a GSD file: /builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd
fl.pyx:279: RuntimeError
_________________ ERROR at setup of TestGSDParser.test_tt_size _________________
[gw0] linux2 -- Python 2.7.14 /usr/bin/python2
self = <MDAnalysisTests.topology.test_gsd.TestGSDParser object at 0x7fff9f742590>
filename = '/builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd'
    @pytest.fixture()
    def top(self, filename):
        with self.parser(filename) as p:
>           yield p.parse()
topology/base.py:46: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/topology/GSDParser.py:107: in parse
    with gsd.hoomd.open(self.filename,mode='rb') as t :
/usr/lib64/python2.7/site-packages/gsd/hoomd.py:799: in open
    schema_version=[1,2]);
fl.pyx:163: in gsd.fl.open
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: Not a GSD file: /builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd
fl.pyx:279: RuntimeError
________________ ERROR at setup of TestGSDParser.test_attr_size ________________
[gw0] linux2 -- Python 2.7.14 /usr/bin/python2
self = <MDAnalysisTests.topology.test_gsd.TestGSDParser object at 0x7fffa12d0890>
filename = '/builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd'
    @pytest.fixture()
    def top(self, filename):
        with self.parser(filename) as p:
>           yield p.parse()
topology/base.py:46: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/topology/GSDParser.py:107: in parse
    with gsd.hoomd.open(self.filename,mode='rb') as t :
/usr/lib64/python2.7/site-packages/gsd/hoomd.py:799: in open
    schema_version=[1,2]);
fl.pyx:163: in gsd.fl.open
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: Not a GSD file: /builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd
fl.pyx:279: RuntimeError
=================================== FAILURES ===================================
__________________________ TestGSDParser.test_output ___________________________
[gw0] linux2 -- Python 2.7.14 /usr/bin/python2
self = <MDAnalysisTests.topology.test_gsd.TestGSDParser object at 0x7fff9ad6b810>
filename = '/builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd'
    def test_output(self, filename):
        """Testing the call signature"""
        with self.parser(filename) as p:
>           top = p.parse()
topology/base.py:51: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/topology/GSDParser.py:107: in parse
    with gsd.hoomd.open(self.filename,mode='rb') as t :
/usr/lib64/python2.7/site-packages/gsd/hoomd.py:799: in open
    schema_version=[1,2]);
fl.pyx:163: in gsd.fl.open
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: Not a GSD file: /builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd
fl.pyx:279: RuntimeError
_____________________ TestGSDParser.test_creates_universe ______________________
[gw0] linux2 -- Python 2.7.14 /usr/bin/python2
self = <MDAnalysisTests.topology.test_gsd.TestGSDParser object at 0x7fff9580c890>
filename = '/builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd'
    def test_creates_universe(self, filename):
        """Check that Universe works with this Parser"""
>       u = mda.Universe(filename)
topology/base.py:94: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/core/universe.py:261: in __init__
    self._topology = p.parse(**kwargs)
../../../../BUILDROOT/python-MDAnalysis-0.17.0-1.fc29.ppc64/usr/lib64/python2.7/site-packages/MDAnalysis/topology/GSDParser.py:107: in parse
    with gsd.hoomd.open(self.filename,mode='rb') as t :
/usr/lib64/python2.7/site-packages/gsd/hoomd.py:799: in open
    schema_version=[1,2]);
fl.pyx:163: in gsd.fl.open
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   RuntimeError: Not a GSD file: /builddir/build/BUILD/MDAnalysis-0.17.0/MDAnalysisTests-0.17.0/MDAnalysisTests/data/example.gsd
fl.pyx:279: RuntimeError
 2 failed, 6182 passed, 19 skipped, 1 xfailed, 18542 warnings, 10 error in 354.73 seconds

Code to reproduce the behaviour

pytest-2.7 -v topology/test_gsd.py

I reported this upstream and it looks like big-endian arches are not supported. Could GSD be made optional again (like in 0.16?)

Currently version of MDAnalysis:

0.17.0

Mar 13 '18 14:03 rathann

@rathann just so I understand your preferred solution: gsd is not supporting some of the arch that you are packaging for and because gsd is a required dependency for MDAnalysis you would not be able to ship MDAnalysis without breaking dependencies?

Thus you would like gsd to become an optional dependency for MDAnalysis, i.e., it should be possible to import the library without a problem when gsd is missing and only get an error if one wants to use specific gsd functionality?

Mar 14 '18 01:03 orbeckst

Yes, if that's possible. Otherwise I'll probably have to stop shipping MDAnalysis built for big-endian arches.

Mar 14 '18 12:03 rathann

Can we flag gsd tests as known failures on big endian? Ie can we detect this from inside python.

On Wed, 14 Mar 2018, 12:19 rathann, [email protected] wrote:

Yes, if that's possible. Otherwise I'll probably have to stop shipping MDAnalysis built for big-endian arches.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/MDAnalysis/mdanalysis/issues/1829#issuecomment-373001247, or mute the thread https://github.com/notifications/unsubscribe-auth/AI0jB4xYeaA57KK1xnATBtivJ_vFUUOuks5teQq6gaJpZM4So0tr .

Mar 14 '18 12:03 richardjgowers

Tests are one thing, but the other would be if @rathann ships builds of MDAnalysis that are known to be broken. That would be bad and I don't think anybody wants that.

I would like MDA to be useable on most HPC so big-endian builds should be possible.

If we make gsd optional again then we need to protect the gsd import --- perhaps at the format registration stage? For the tests we can just skip if gsd is not present.

-- Oliver Beckstein email: [email protected]

Am Mar 14, 2018 um 05:56 schrieb Richard Gowers [email protected]:

Can we flag gsd tests as known failures on big endian? Ie can we detect this from inside python.

Mar 14 '18 16:03 orbeckst

@orbeckst yeah - we could not import GSD on that architecture. Something like:

# coordinates/__init__.py
if not BIG_ENDIAN:
    from . import GSD
else:
    warnings.warn('GSD not available because of incompatibility')

This should make MDA blissfully unaware of GSD as a format keyword.

But with this the install process could be the same for both.

If we made gsd optional as a requirement then it wouldn't be installed by default, which means you'd get annoying error messages put in by us like "you've not got gsd, do pip install gsd".

Mar 14 '18 17:03 richardjgowers

I would just do a try: import gsd; except ImportError: ... instead of being specific about BIG_ENDIAN.

pip

Requiring gsd in setup.py will prevent users on big-endian to install all of MDA because the requirements cannot be fulfilled. Perhaps we can change the requirements list in setup.py after detecting architecture? Then pip would still function.

conda

I don't know about conda packages – can they be made architecture-aware (@kain88-de ?).

Managing dependencies...

I don't like tying MDAnalysis to other supporting packages that might not be supported as well as the main dependencies or MDA itself. We should think of a way to mitigate these problems.

In this case I am rather prepared to downgrade gsd to optional than to compromise platform availability. If this means that users of gsd get an annoying message that they need to conda install gsd or pip install gsd then I think that's the price to pay for using a package that is only partially supported. And the vast majority of our users will see no difference.

Mar 14 '18 17:03 orbeckst

@orbeckst I think you can install gsd on the big endian systems though, it just doesn't work. @rathann is this right?

Mar 14 '18 17:03 richardjgowers

Yes, that's my understanding, but I don't think @rathann wants to ship a package that is known to be non-functional on the arch. (I wouldn't)

Mar 14 '18 17:03 orbeckst

By the way, for users to install every dependency we can use

pip install MDAnalysis[all]

For instance, right now using [analysis,amber] will additionally install netCDF4 (for fast NCDF writing of amber trajectories), and seaborn and sklearn for things in analysis. We can make all just install every additional dependency including gsd.

My main issue is then what to do with conda packages. We could make that it installs a broken gsd as a dependency (the conda package of gsd will happily install itself) and then check ourselves if this is a broken gsd, disable it. When gsd raises its own error then we can test for that.

Mar 14 '18 17:03 orbeckst

The thing is that there are many ways to install a package. Even if it is not available as a RPM because @rathann knowns it does not work, it can still be installed with pip. Also, just basing the error message on if GSD is installed or not means the message will state that the package is not available (pushing the user to install a package that is broken for its arch) and not that it will not work.

Mar 14 '18 18:03 jbarnoud

The way to implement @richardjgowers suggestion seems to be:

BIG_ENDIAN = (sys.byteorder == 'big')

Mar 14 '18 18:03 jbarnoud

conda-forge doesn't support 32bit or distinguishes between big/little endian
We do not know if GSD doesn't work on big-endian
GSD cannot read files written in little-endian when running on big-endian this causes our tests to fail when we read a little-endian file from the test suite on a big-endian system.

Preparing an alternative big-endian file that is read on the appropriate platform might be an easier solution. @rathann could start with a patch for the package system and we can then look to include it.

Mar 15 '18 08:03 kain88-de

We do not know if GSD doesn't work on big-endian

GSD cannot read files written in little-endian when running on big-endian this causes our tests to fail when we read a little-endian file from the test suite on a big-endian system.

is what I call "doesn't work". Using a file generated on big-endian for testing on big-endian and pretending everything is fine doesn't cut it in my opinion, as real world cross-platform file compatibility is still compromised.

Mar 15 '18 13:03 rathann

@richardjgowers

I think you can install gsd on the big endian systems though, it just doesn't work. @rathann is this right?

It doesn't work in the way @kain88-de described above.

Mar 15 '18 13:03 rathann

as real world cross-platform file compatibility is still compromised.

According to the author, GSD is not cross-platform compatible. Since the format isn't intended for sharing trajectories. Making the library an optional dependency won't change that. The question is if one can still read/write files when staying on the same computer, independent of its architecture.

Mar 15 '18 13:03 kain88-de

Thanks for the explanation @kain88-de. So the first question to answer is if gsd can be built successfully on big-endian and if it is able to process GSD trajectories on big-endian. (In our case "process" would mean to be able to read a gsd trajectory that was produced on big-endian and write a trajectory that can also be read on the same machine.)

If this is the case then I am coming around to the idea for MDA to have a little-endian and a big-endian test trajectory and we would then at test-time decide which one to present to our GSD tests. (We might even be able to have our GSDReader check if a trajectory was produced on big- or little endian and then fail gracefully, at least until GSD itself implements the check.)

Does anyone have access to a big-endian machine to

build GSD
produce a big-endian test trajectory?

Mar 19 '18 01:03 orbeckst

@orbeckst I have a ppc64 VM and can do it, but I'm going to need some step by step instructions for producing a trajectory.

Mar 19 '18 09:03 rathann

I don't know how to create a native GSD trajectory, @rcortini produced the original test trajectory – can you chime in?

MDAnalysis does not have a GSD writer but one could use the gsd.hoomd library directly.

But before generating some random trajectories I'd rather investigate if there's a way to re-create our example.gsd. @rcortini do you have an idea how we could do this? Something like

on little-endian: extract data from example.gsd (eg to ASCII file)
on big-endian: recreate trajectory with gsd library

Mar 24 '18 03:03 orbeckst

@rcortini can you briefly tell us how you created the HOOMD gsd trajectory that we use as a test file in MDAnalysis? Thanks!

Apr 06 '18 01:04 orbeckst

Hi, sorry for the late response.

Here is a piece of code that will produce an almost identical trajectory. It requires the installation of the HOOMD-blue python package.

from __future__ import division
import hoomd
import hoomd.md

# init HOOMD-blue
hoomd.context.initialize("");

# init particles in the system
uc = hoomd.lattice.unitcell(N = 1,
                            a1 = [10.8, 0,   0],
                            a2 = [0,    1.2, 0],
                            a3 = [0,    0,   1.2],
                            dimensions = 3,
                            position = [[0,0,0]],
                            type_name = ['R'],
                            mass = [1.0],
                            moment_inertia = [[0,
                                               1/12*1.0*8**2,
                                               1/12*1.0*8**2]],
                            orientation = [[1, 0, 0, 0]]);
system = hoomd.init.create_lattice(unitcell=uc, n=[2,18,18]);
system.particles.types.add('A');

# define rigid bodies
rigid = hoomd.md.constrain.rigid();
rigid.set_param('R', 
                types=['A']*8,
                positions=[(-4,0,0),(-3,0,0),(-2,0,0),(-1,0,0),
                           (1,0,0),(2,0,0),(3,0,0),(4,0,0)]);
rigid.create_bodies()

# define particle interactions
nl = hoomd.md.nlist.cell()
lj = hoomd.md.pair.lj(r_cut=2**(1/6), nlist=nl)
lj.set_params(mode='shift')
lj.pair_coeff.set(['R', 'A'], ['R', 'A'], epsilon=1.0, sigma=1.0)

# define integrator
hoomd.md.integrate.mode_standard(dt=0.005)
rigid = hoomd.group.rigid_center();
hoomd.md.integrate.langevin(group=rigid, kT=1.0, seed=42);

# specify trajectory output
hoomd.dump.gsd("example.gsd",
               period=2e3,
               group=hoomd.group.all(),
               overwrite=True);     

# run simulation
hoomd.run(4e3);

This is basically the example of the "Rigid Rods" in the hoomd-examples package.

Apr 06 '18 09:04 rcortini

Many thanks. (Sorry for the long delay.)

I think we should pick up this issue for 0.19.x.

[ ] create big-endian test trajectories
[ ] teach our GSD test to choose the correct trajectory

Oct 09 '18 20:10 orbeckst

This sounds like an upstream thing, we don't actually read the GSD file directly

Oct 09 '18 21:10 richardjgowers

Oh I see, we want two different gsd files to test...

Oct 09 '18 21:10 richardjgowers

FWIW, it is now possible to test the s390x big-endian architecture in Travis CI. The manylinux 2014 standard describes support for binary wheels with ARMv8, ppc64le, and s390x architectures among others.

Dec 13 '19 05:12 tylerjereddy

@tylerjereddy how does one do big-endian tests on Travis CI? Do you have a link to docs? – Are you referring to the Arm64 support https://blog.travis-ci.com/2019-10-07-multi-cpu-architecture-support ?

Dec 13 '19 17:12 orbeckst

See here: https://blog.travis-ci.com/2019-11-12-multi-cpu-architecture-ibm-power-ibm-z

The NumPy PR doing this a few weeks ago: https://github.com/numpy/numpy/pull/14907

Note that we may have mixed results considering we are much higher up the ecosystem pyramid (some of our deps may have to be built from source each time on that arch until the conda/pypi binaries become available with newer standards, etc.; aggressive caching might alleviate that after the initial build, maybe)

Dec 13 '19 20:12 tylerjereddy

Confirming tests are still failing in v2.7.0 on a big-endian system (s390x). See https://ci.debian.net/packages/m/mdanalysis/testing/s390x/50289858/

Debian has an s390x porterbox. You could request access if you'd like to dig into it more closely.

Aug 09 '24 09:08 drew-parsons

Thanks @drew-parsons

It might be worth mentioning that GSD is no longer a required component of MDAnalysis, if it helps get a Debian package out, I would suggest not including it as an optional dependency.

Aug 09 '24 22:08 IAlibay

True. I've already deactivated gsd for s390x so the current errors I'm seeing are something else. Perhaps I should file a separate bug report, but I was thinking to keep the big-endian (or at least s390x) errors in one place. The summary of errors I'm referring to is

692s FAILED testsuite/MDAnalysisTests/analysis/test_base.py::test_start_stop_step[run_kwargs0-frames0]
692s FAILED testsuite/MDAnalysisTests/analysis/test_base.py::test_start_stop_step[run_kwargs1-frames1]
692s FAILED testsuite/MDAnalysisTests/analysis/test_base.py::test_start_stop_step[run_kwargs2-frames2]
692s FAILED testsuite/MDAnalysisTests/analysis/test_base.py::test_start_stop_step[run_kwargs3-frames3]
692s FAILED testsuite/MDAnalysisTests/analysis/test_hydrogenbonds_analysis.py::TestHydrogenBondAnalysisTIP3P::test_count_by_time
692s FAILED testsuite/MDAnalysisTests/analysis/test_hydrogenbonds_analysis.py::TestHydrogenBondAnalysisTIP3P_GuessAcceptors_GuessHydrogens_UseTopology_::test_count_by_time
692s FAILED testsuite/MDAnalysisTests/analysis/test_hydrogenbonds_analysis.py::TestHydrogenBondAnalysisTIP3PStartStep::test_count_by_time
692s FAILED testsuite/MDAnalysisTests/analysis/test_rms.py::TestRMSD::test_rmsd - ...
692s FAILED testsuite/MDAnalysisTests/analysis/test_rms.py::TestRMSD::test_rmsd_frames
692s FAILED testsuite/MDAnalysisTests/analysis/test_rms.py::TestRMSD::test_rmsd_unicode_selection
692s FAILED testsuite/MDAnalysisTests/analysis/test_rms.py::TestRMSD::test_rmsd_single_frame
692s FAILED testsuite/MDAnalysisTests/analysis/test_rms.py::TestRMSD::test_mass_weighted
692s FAILED testsuite/MDAnalysisTests/analysis/test_rms.py::TestRMSD::test_custom_weighted
692s FAILED testsuite/MDAnalysisTests/analysis/test_rms.py::TestRMSD::test_custom_weighted_list
692s FAILED testsuite/MDAnalysisTests/analysis/test_rms.py::TestRMSD::test_rmsd_group_selections
692s FAILED testsuite/MDAnalysisTests/analysis/test_rms.py::TestRMSD::test_rmsd_backbone_and_group_selection
692s FAILED testsuite/MDAnalysisTests/analysis/test_rms.py::TestRMSD::test_ref_mobile_mass_mismapped
692s FAILED testsuite/MDAnalysisTests/coordinates/test_chainreader.py::TestChainReader::test_time
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::TestDCDReader::test_dt
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::TestDCDReader::test_total_time
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::TestDCDReader::test_iter
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::TestDCDReader::test_last_frame
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::TestDCDReader::test_frame_jump
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::TestDCDReader::test_next_gives_second_frame
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::TestDCDReader::test_set_time
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::test_write_istart[0]
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::test_write_istart[1]
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::test_write_istart[2]
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::test_write_istart[37]
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::test_write_istart[None]
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::TestDCDWriter::test_write_trajectory_atomgroup
692s FAILED testsuite/MDAnalysisTests/coordinates/test_dcd.py::TestDCDWriter::test_write_trajectory_universe
692s FAILED testsuite/MDAnalysisTests/coordinates/test_lammps.py::TestLAMMPSDCDReader::test_dt
692s FAILED testsuite/MDAnalysisTests/coordinates/test_lammps.py::TestLAMMPSDCDReader::test_Timestep_time
692s FAILED testsuite/MDAnalysisTests/coordinates/test_memory.py::TestMemoryReader::test_rename_aux
692s FAILED testsuite/MDAnalysisTests/coordinates/test_memory.py::TestMemoryReader::test_iter_as_aux_highf
692s FAILED testsuite/MDAnalysisTests/coordinates/test_memory.py::TestMemoryReader::test_iter_as_aux_lowf
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_coordinates
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_distances
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_dt
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_frame
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_frame_index_0
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_frame_index_1_raises_IndexError
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_full_slice
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_get_writer_from_reader
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_last_slice
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_load_file
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_n_atoms
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_n_frames
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_numres
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_parse_n_atoms
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_pickle_singleframe_reader
692s FAILED testsuite/MDAnalysisTests/coordinates/test_namdbin.py::TestNAMDBINReader::test_time
692s ERROR testsuite/MDAnalysisTests/coordinates/test_copying.py::test_reader_n_atoms[ref_reader25]
692s ERROR testsuite/MDAnalysisTests/coordinates/test_copying.py::test_reader_filename[ref_reader25]
692s ERROR testsuite/MDAnalysisTests/coordinates/test_copying.py::test_reader_copied_extra_attributes[ref_reader_extra_args25]
692s ERROR testsuite/MDAnalysisTests/coordinates/test_copying.py::test_reader_independent_iteration[ref_reader25]
692s ERROR testsuite/MDAnalysisTests/coordinates/test_copying.py::test_reader_initial_frame_maintained[ref_reader25]
692s ERROR testsuite/MDAnalysisTests/coordinates/test_copying.py::test_reader_initial_next[ref_reader25]
692s ERROR testsuite/MDAnalysisTests/coordinates/test_copying.py::test_timestep_copied[ref_reader25]
692s ERROR testsuite/MDAnalysisTests/coordinates/test_copying.py::test_positions_share_memory[ref_reader25]
692s ERROR testsuite/MDAnalysisTests/coordinates/test_namdbin.py::NAMDBINWriter::test_write_different_box
692s ERROR testsuite/MDAnalysisTests/coordinates/test_namdbin.py::NAMDBINWriter::test_write_trajectory_atomgroup
692s ERROR testsuite/MDAnalysisTests/coordinates/test_namdbin.py::NAMDBINWriter::test_write_trajectory_universe
692s ERROR testsuite/MDAnalysisTests/coordinates/test_namdbin.py::NAMDBINWriter::test_write_selection
692s ERROR testsuite/MDAnalysisTests/coordinates/test_namdbin.py::NAMDBINWriter::test_write_not_changing_ts
692s = 53 failed, 18525 passed, 635 skipped, 8 xfailed, 1 xpassed, 8121 warnings, 13 errors in 574.03s (0:09:34) =

Aug 10 '24 00:08 drew-parsons