Fix Skipped Unit tests

[dkeeney]

There are Unit tests that have been skipped in order to complete the big pybind11 PR. We now need to go back and fix them.

[breznak]

Please review PR #169 before this, as it removes some of the skipped tests. (our py.test coverage is not that good!, seems most of the tests are for Sparse*Matrix that is unused & removed, normal classes with bindings lack the tests)

[dkeeney]

Somewhat related to the skipped Unit tests....
I am trying to build using C++17 on Ubuntu. C++17 uses a little different code for the selector function select2nd( ) and it breaks the functor struct predicate_compose.

select2nd( ) is at nupic/math/Utils.hpp:390

In bindings/math/py_Math.cpp:114 we generate the .py bindings for winnerTakesAll_3( ). This calls nupic::winnerTakesAll3() which eventually makes a call to nupic/math/Math.hpp:776 where it instantiates the functor struct predicate_compose. There is an error setting the type of 'argument_type'. I cannot figure out how to do that.

template <typename O, typename S> struct predicate_compose {
  typedef typename S::argument_type argument_type;

  O o;  // operation function object
  S s;  // selection function object

  inline bool operator()(const argument_type &x,  const argument_type &y) const {
    return o(s(x), s(y));
  }
};

Meanwhile, I commented out winnerTakesAll_3 in the bindings.

[ctrl-z-9000-times]

Here is a list of all of the skipped tests:

In python2 only
cast_mode_test.py::TestCastMode::testCastMode <- ../../../../../usr/lib/python2.7/unittest/case.py SKIPPED

Both python2 & python3:
network_test.py::NetworkTest::testParameters SKIPPED
network_test.py::NetworkTest::testSerializationWithPyRegion SKIPPED
nupic_random_test.py::TestNupicRandom::testEquals SKIPPED
nupic_random_test.py::TestNupicRandom::testNupicRandomPickling SKIPPED
nupic_random_test.py::TestNupicRandom::testSampleBadDtype SKIPPED
nupic_random_test.py::TestNupicRandom::testSampleDifferentDtypes SKIPPED
nupic_random_test.py::TestNupicRandom::testSamplePopulationTooSmall SKIPPED
nupic_random_test.py::TestNupicRandom::testSampleSequenceRaisesTypeError SKIPPED
nupic_random_test.py::TestNupicRandom::testSampleWrongDimensionsChoices SKIPPED
nupic_random_test.py::TestNupicRandom::testSampleWrongDimensionsPopulation SKIPPED
nupic_random_test.py::TestNupicRandom::testShuffleBadDtype SKIPPED
nupic_random_test.py::TestNupicRandom::testShuffleEmpty2 SKIPPED
sparse_binary_matrix_test.py::UnitTests::testScalability <- ../../../../../usr/lib/python2.7/unittest/case.py SKIPPED
sparse_link_test.py::SparseLinkTest::testDenseToDense SKIPPED
sparse_link_test.py::SparseLinkTest::testDenseToDenseToDenseDelay SKIPPED
sparse_link_test.py::SparseLinkTest::testDenseToDenseToSparseDelay SKIPPED
sparse_link_test.py::SparseLinkTest::testDenseToSparse SKIPPED
sparse_link_test.py::SparseLinkTest::testSparseToDense SKIPPED
sparse_link_test.py::SparseLinkTest::testSparseToSparse SKIPPED
sparse_link_test.py::SparseLinkTest::testSparseToSparseToDenseDelay SKIPPED
sparse_link_test.py::SparseLinkTest::testSparseToSparseToSparseDelay SKIPPED
sparse_matrix_test.py::SparseMatrixTest::test_LogSumApprox <- ../../../../../usr/lib/python2.7/unittest/case.py SKIPPED
sparse_matrix_test.py::SparseMatrixTest::test_construction SKIPPED
sparse_matrix_test.py::SparseMatrixTest::test_getstate_setstate SKIPPED
sparse_matrix_test.py::SparseMatrixTest::test_rightVecProd SKIPPED
sparse_matrix_test.py::SparseMatrixTest::test_setBox SKIPPED
sparse_matrix_test.py::SparseMatrixTest::test_test_nta_set SKIPPED
algorithms/cells4_test.py::Cells4Test::testEquals SKIPPED
algorithms/cells4_test.py::Cells4Test::testLearn SKIPPED

I think we should prioritize fixing these as follows:

1. Network tests
2. Cells4 tests
3. Random tests
4. Sparse-matrix and sparse-link tests. These might be removed, see #169
5. Python2 only tests.

[dkeeney]

The networkTest tests are .py serialization. I think that should be on my plate.

This involves implementing a save( ) and load( ) in PyBindRegion.cpp by initiating a pickle on the python code. It is coded but does not work. This does need to be fixed but probably not a priority 1 thing. Everything but serialization should work.

I think MSVC is probably a little higher priority but I will go back to this pickle thing after MSVC is working.

What is maybe not in that list is the fact that our test coverage is fairly thin. I keep thinking that there should be more tests someplace. If you happen to be wandering around in the nupic repository (the .py code) keep an eye out for unit tests that we don't have.

[ctrl-z-9000-times]

I think I found the unit tests of which you speak. They're at: htm-community/nupic.py/tests

Some of them use nupic and others use nupic.bindings

[dkeeney]

Cool. Maybe we can incorporate some of those in our testbed. 👍

[ctrl-z-9000-times]

Update:

We have removed several of the tests since the underlying code was pruned away. Note: many of these test cases might be for things which we've removed, in which case the test case should be removed as well. Here is an updated list of tests left to be fixed:

• network_test.py::NetworkTest::testParameters SKIPPED
• ~nupic_random_test.py::TestNupicRandom::testEquals SKIPPED~ PR #506
• ~nupic_random_test.py::TestNupicRandom::testSampleBadDtype SKIPPED~ PR #506
• ~nupic_random_test.py::TestNupicRandom::testSampleDifferentDtypes SKIPPED~ Test not found
• ~nupic_random_test.py::TestNupicRandom::testSamplePopulationTooSmall SKIPPED~ PR #506
• ~nupic_random_test.py::TestNupicRandom::testSampleSequenceRaisesTypeError SKIPPED~ PR #506
• ~nupic_random_test.py::TestNupicRandom::testSampleWrongDimensionsChoices SKIPPED~ Test not found
• ~nupic_random_test.py::TestNupicRandom::testSampleWrongDimensionsPopulation SKIPPED~ PR #506
• ~nupic_random_test.py::TestNupicRandom::testShuffleBadDtype SKIPPED~ PR #506
• ~nupic_random_test.py::TestNupicRandom::testShuffleEmpty2 SKIPPED~ PR #506

[Zbysekz]

Hello, i am pretty new here, but i will try to fix these skipped tests, or at least figure out what is happening.

[dkeeney]

That would be wonderful. Thanks.

[breznak]

hi @Zbysekz, thank you very much for your help! Let us know if we can help you anyhow during your progress. cheers :wave:

[Zbysekz]

Can i ask for your advice? I have two questions:

1. When working on tests (without C++ IDE) should i use "python setup.py install --user" to recompile everything or is there any faster way? Considering that i was working on py bindings. Also when i am using eclipse how can i rebuild everything from IDE? Should i define command to call setup or use for example make ?

2. How to run coverage tests? I see that in folder bindings/py/tests/htmlcov are some results but it seems that running "python setup.py test" does not generate these reports.

[breznak]

i use "python setup.py install --user" to recompile everything or is there any faster way?

assuming you're changing only the PY code, that is what I do. First compile takes long, as it recompiles the c++, later changes should be minimal

lso when i am using eclipse how can i rebuild everything from IDE?

I'm using netbeans, I open it as CMake project, so Re/Build for c++ works out of the box For python, I again just manually re-run the setup.py

How to run coverage tests? I

there's example in one of the CIs (Travis?), but coverage is not ran, so no guarantees of how/if it works

[ctrl-z-9000-times]

is there any faster way?

Yes. The python installation process will use whatever compiled files it can find in the build directory. So I compile the C++ separately for greater control, with the following linux commands:

pushd build/scripts
cmake ../.. -DCMAKE_INSTALL_PREFIX=../Release -DCMAKE_BUILD_TYPE=Release
make VERBOSE=1 install -j 6
popd
py3 setup.py install --user --force test

Notice that the make command is given the flag -j 6 which tells it to use 6 separate processes to compile 6 files at a time.

Also notice that I set the install prefix manually, you need this if you want to change the build type from Release to Debug. Otherwise the python installation process will be unable to find the debug-version of the library.

And finally notice the python install flag --force which is needed because otherwise python will not overwrite any existing files with new versions of the library (because nupic's version number does not increase).

[breznak]

Are there still some py, bindings tests that are TBD?