Many tests fail on FreeBSD

[yurivict]

26% tests passed, 84 tests failed out of 113

The following tests FAILED:
	 11 - publishing_single_path_resources_http_get_acceptance_test_suite (OTHER_FAULT)
	 12 - publishing_single_path_resources_http_put_acceptance_test_suite (OTHER_FAULT)
	 13 - publishing_single_path_resources_http_head_acceptance_test_suite (OTHER_FAULT)
	 14 - publishing_single_path_resources_http_post_acceptance_test_suite (OTHER_FAULT)
	 15 - publishing_single_path_resources_http_trace_acceptance_test_suite (OTHER_FAULT)
         ...

Example failures:

# ./work/restbed-4.6/test/regression/uncaught_exception_when_peer_closes_connection_regression_test_suite

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
uncaught_exception_when_peer_closes_connection_regression_test_suite is a Catch v1.9.6 host application.
Run with -? for options

-------------------------------------------------------------------------------
peer closes connection without sending data
-------------------------------------------------------------------------------
test/regression/source/uncaught_exception_when_peer_closes_connection.cpp:55
...............................................................................

test/regression/source/uncaught_exception_when_peer_closes_connection.cpp:55: FAILED:
due to a fatal error condition:
  SIGABRT - Abort (abnormal termination) signal

===============================================================================
test cases: 1 | 1 failed
assertions: 1 | 1 failed

Abort trap

# ./work/restbed-4.6/test/regression/path_parameters_off_by_one_regression_test_suite

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
path_parameters_off_by_one_regression_test_suite is a Catch v1.9.6 host application.
Run with -? for options

-------------------------------------------------------------------------------
single path parameter with root are off by one
-------------------------------------------------------------------------------
test/regression/source/path_parameters_off_by_one.cpp:41
...............................................................................

test/regression/source/path_parameters_off_by_one.cpp:66: FAILED:
  REQUIRE( 200 == response->get_status_code( ) )
with expansion:
  200 == 0

test/regression/source/path_parameters_off_by_one.cpp:41: FAILED:
  {Unknown expression after the reported line}
with expansion:

due to a fatal error condition:
  SIGABRT - Abort (abnormal termination) signal

===============================================================================
test cases: 1 | 1 failed
assertions: 2 | 2 failed

Abort trap

[ben-crowhurst]

Please state your environment compiler, OS, etc...

[yurivict]

FreeBSD 11.1 amd64, clang-4.0.0.

[ben-crowhurst]

Thanks, attempting a build now.

[yurivict]

Here's the port. It builds fine. If you can't reproduce on linux, the easiest is to build the port in the FreeBSD VM.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220912

[ben-crowhurst]

So I've attempted to build on FreeBSD 11 AMD64 with clang-3.8.0 with no errors reported or tests failing.

Are you able to supply anymore information about the issues you see?

[yurivict]

I just reran build and tests with clang-3.8 and still the same.

Here is another failing case output:

# ./work/restbed-4.6/test/acceptance/custom_service_error_handling_acceptance_test_suite

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
custom_service_error_handling_acceptance_test_suite is a Catch v1.9.6 host application.
Run with -? for options

-------------------------------------------------------------------------------
Scenario: custom service error handler
     Given: I publish a faulty resources and set a service error handler
      When: I perform a HTTP 'GET' request to '/resources/1'
      Then: I should see a '0' (No Appropriate Status Message Found) status
            code
-------------------------------------------------------------------------------
test/acceptance/source/custom_error_handling/service.cpp:76
...............................................................................

test/acceptance/source/custom_error_handling/service.cpp:79: FAILED:
  REQUIRE( "No Appropriate Status Message Found" == response->get_status_message( ) )
with expansion:
  "No Appropriate Status Message Found"
  ==
  "Error"

test/acceptance/source/custom_error_handling/service.cpp:76: FAILED:
  {Unknown expression after the reported line}
with expansion:

due to a fatal error condition:
  SIGABRT - Abort (abnormal termination) signal

===============================================================================
test cases: 1 | 1 failed
assertions: 3 | 1 passed | 2 failed

Abort trap

I traced its syscalls with ktrace, here is a suspicious place:

 41704 custom_service_erro CALL  connect(0xa,0x7fffdfffcac0,0x10)
 41704 custom_service_erro STRU  struct sockaddr { AF_INET, 127.0.0.1:1984 }
 41704 custom_service_erro RET   connect -1 errno 36 Operation now in progress

The testcase attempts to connect socket with fd=0xa, gets "Operation now in progress", which means that it needs to try again later, and it never tries again as it should.

Are you sure custom_service_error_handling_acceptance_test_suite passes in all 3 cases for you?

[ben-crowhurst]

I'm seeing green across the board. I'll upgrade my version of clang and see if I can't replicate the problems.

[ben-crowhurst]

Please provide the exact build arguments in use. I'm having a hell of time replicating this issue, which is alarming.

[yurivict]

Here is how to reproduce:

1. cd /usr/ports
2. fetch https://bz-attachments.freebsd.org/attachment.cgi?id=184583 -o restbed.shar
3. sh restbed.shar
4. cd www/restbed
5. make test
6. observe test failures

[ben-crowhurst]

Thank you! I will test this scenario and give appropriate feedback in the next hour.

[ben-crowhurst]

Please note you mention the license is GPL, this is incorrect. We have adopted the AGPL.

[ben-crowhurst]

The archive you supplied fails with:

c - www/restbed
Ambiguous output redirect.

[ben-crowhurst]

Have you attempted to build the project as per the README instructions? If so, what was the result?

[yurivict]

Please note you mention the license is GPL, this is incorrect. We have adopted the AGPL.

Thanks, will correct it.

Ambiguous output redirect.

Such message should not occur from the above commands.

Have you attempted to build the project as per the README instructions? If so, what was the result?

Just tried, same result.

[ben-crowhurst]

Following the updated instructions results in:

The Corvusoft:asio-dependency:f5c5708:a/dependency/asio GH_TUPLE line has
a tag containing something else then [a-zA-Z0-9_]
*** Error code 1

Stop.
make: stopped in /usr/ports/www/restbed

[yurivict]

Instructions as a shell script. If fails for me with the messages from my OP.

#!/bin/sh

cd /usr/ports &&
rm -rf www/restbed
fetch https://bz-attachments.freebsd.org/attachment.cgi?id=184583 -o restbed.shar &&
sh restbed.shar &&
cd www/restbed &&
make test

[ben-crowhurst]

root@freebsd: ~ # ./install.sh
fetch: https://bz-attachments.freebsd.org/attachment.cgi?id=184583: size of remote file is not known
restbed.shar                                                            4209  B      10  kBps 00m00s
c - www/restbed
x - www/restbed/distinfo
x - www/restbed/Makefile
x - www/restbed/pkg-descr
x - www/restbed/pkg-plist
The Corvusoft:asio-dependency:f5c5708:a/dependency/asio GH_TUPLE line has a tag containing something else than [a-zA-Z0-9_]
*** Error code 1

Stop.
make: stopped in /usr/ports/www/restbed
root@freebsd: ~ # uname -a
FreeBSD freebsd 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep 29 01:43:23 UTC 2016 [email protected]:/usr/obj/usr/src/sys/GENERIC amd64
root@freebsd: ~ #

[ben-crowhurst]

Can I ask you to confirm the following?

git clone --recursive https://github.com/corvusoft/restbed.git
mkdir restbed/build
cd restbed/build
cmake -DBUILD_TESTS=YES -DBUILD_EXAMPLES=YES ..
make install
make test

If you receive errors please provide as much information as possible.

[yurivict]

If you receive test errors please provide as much information as possible.

Exactly the same errors as before.

What's the version you are running? What does 'uname -a' say?

[ben-crowhurst]

I provided the output from uname in my previous post.

[ben-crowhurst]

Odd I see no failures when running the README.md build instructions.

[ben-crowhurst]

I'm currently operating on VirtualBox. I'll move to hardware to ensure there is no funny business.

[yurivict]

It's odd that you get this:

The Corvusoft:asio-dependency:f5c5708:a/dependency/asio GH_TUPLE line has a tag containing something else

This was happening with pre-11 versions. In 11 this shouldn't happen.

There should be no funny business with VBox. To save time, I recommend to install 11.1 image from here: https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.1/

[ben-crowhurst]

Ok attempting now.

[ben-crowhurst]

Sorry for the delay. I've managed to install 11.1 and I have received a single test failure.

The following tests FAILED:
               73 - service_connection_timeout_acceptance_test_suite (OTHER_FAULT)

[callum-kirby]

@ben-crowhurst I resolved issue as requested. No other issues found.

[ben-crowhurst]

@yurivict?

[yurivict]

In 9a5c69e, I still get 27% tests passed, 83 tests failed out of 113.

[ben-crowhurst]

Would it be possible to create an ISO image of your environment and make available for download?

[yurivict]

That's what I was also thinking. Is VirtualBox VM image okay?