rbsec
Errors with docker test script
I setup an Ubuntu 22.04 host to do docker tests and I'm getting the following errors when run against rbsec/sslscan@master. @jtesta Do you have any insights into what might be going on here?
Thanks.
Running all tests...
Test #1 passed.
Test #2 passed.
Test #3 passed.
Test #4 passed.
Test #5 passed.
Test #6 passed.
Test #7 passed.
Test #8 passed.
Test #9 skipped.
Test #10 skipped.
Test #11 passed.
Test #12 passed.
Test #13 FAILED.
--- docker_test/expected_output/test_13.txt 2025-01-26 00:19:21.826672304 +0000
+++ /tmp/sslscan_test-results_Qf5TlhAjUz/test_13.txt 2025-01-26 00:26:58.671104306 +0000
@@ -6,8 +6,8 @@
SSL/TLS Protocols:
SSLv2 disabled
SSLv3 disabled
-TLSv1.0 enabled
-TLSv1.1 enabled
+TLSv1.0 disabled
+TLSv1.1 disabled
TLSv1.2 enabled
TLSv1.3 enabled
@@ -15,7 +15,7 @@
Server supports TLS Fallback SCSV
TLS renegotiation:
-Secure session renegotiation supported
+Session renegotiation not supported
TLS Compression:
Compression disabled
@@ -23,8 +23,6 @@
Heartbleed:
TLSv1.3 not vulnerable to heartbleed
TLSv1.2 not vulnerable to heartbleed
-TLSv1.1 not vulnerable to heartbleed
-TLSv1.0 not vulnerable to heartbleed
Supported Server Cipher(s):
Preferred TLSv1.3 128 bits TLS_AES_128_GCM_SHA256 Curve 25519 DHE 253
@@ -49,24 +47,13 @@
Accepted TLSv1.2 128 bits AES128-CCM
Accepted TLSv1.2 256 bits AES256-SHA
Accepted TLSv1.2 128 bits AES128-SHA
-Preferred TLSv1.1 256 bits ECDHE-RSA-AES256-SHA Curve 25519 DHE 253
-Accepted TLSv1.1 256 bits DHE-RSA-AES256-SHA DHE 2048 bits
-Accepted TLSv1.1 128 bits ECDHE-RSA-AES128-SHA Curve 25519 DHE 253
-Accepted TLSv1.1 128 bits DHE-RSA-AES128-SHA DHE 2048 bits
-Accepted TLSv1.1 256 bits AES256-SHA
-Accepted TLSv1.1 128 bits AES128-SHA
-Preferred TLSv1.0 256 bits ECDHE-RSA-AES256-SHA Curve 25519 DHE 253
-Accepted TLSv1.0 256 bits DHE-RSA-AES256-SHA DHE 2048 bits
-Accepted TLSv1.0 128 bits ECDHE-RSA-AES128-SHA Curve 25519 DHE 253
-Accepted TLSv1.0 128 bits DHE-RSA-AES128-SHA DHE 2048 bits
-Accepted TLSv1.0 256 bits AES256-SHA
-Accepted TLSv1.0 128 bits AES128-SHA
Server Key Exchange Group(s):
TLSv1.3 128 bits secp256r1 (NIST P-256)
TLSv1.3 192 bits secp384r1 (NIST P-384)
TLSv1.3 260 bits secp521r1 (NIST P-521)
TLSv1.3 128 bits x25519
+TLSv1.3 224 bits x448
TLSv1.3 112 bits ffdhe2048
TLSv1.3 128 bits ffdhe3072
TLSv1.3 150 bits ffdhe4096
@@ -76,6 +63,7 @@
TLSv1.2 192 bits secp384r1 (NIST P-384)
TLSv1.2 260 bits secp521r1 (NIST P-521)
TLSv1.2 128 bits x25519
+TLSv1.2 224 bits x448
SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
Test #14 FAILED.
--- docker_test/expected_output/test_14.txt 2025-01-26 00:19:21.826672304 +0000
+++ /tmp/sslscan_test-results_Qf5TlhAjUz/test_14.txt 2025-01-26 00:27:01.675104757 +0000
@@ -25,21 +25,21 @@
TLSv1.2 not vulnerable to heartbleed
Supported Server Cipher(s):
-Preferred TLSv1.3 128 bits TLS_AES_128_GCM_SHA256 Curve P-521 DHE 521
-Accepted TLSv1.3 256 bits TLS_AES_256_GCM_SHA384 Curve P-521 DHE 521
-Accepted TLSv1.3 256 bits TLS_CHACHA20_POLY1305_SHA256 Curve P-521 DHE 521
-Accepted TLSv1.3 128 bits TLS_AES_128_CCM_SHA256 Curve P-521 DHE 521
-Preferred TLSv1.2 256 bits ECDHE-RSA-AES256-GCM-SHA384 Curve P-521 DHE 521
+Preferred TLSv1.3 128 bits TLS_AES_128_GCM_SHA256 Curve 448 DHE 448
+Accepted TLSv1.3 256 bits TLS_AES_256_GCM_SHA384 Curve 448 DHE 448
+Accepted TLSv1.3 256 bits TLS_CHACHA20_POLY1305_SHA256 Curve 448 DHE 448
+Accepted TLSv1.3 128 bits TLS_AES_128_CCM_SHA256 Curve 448 DHE 448
+Preferred TLSv1.2 256 bits ECDHE-RSA-AES256-GCM-SHA384 Curve 448 DHE 448
Accepted TLSv1.2 256 bits DHE-RSA-AES256-GCM-SHA384 DHE 8192 bits
-Accepted TLSv1.2 256 bits ECDHE-RSA-CHACHA20-POLY1305 Curve P-521 DHE 521
+Accepted TLSv1.2 256 bits ECDHE-RSA-CHACHA20-POLY1305 Curve 448 DHE 448
Accepted TLSv1.2 256 bits DHE-RSA-CHACHA20-POLY1305 DHE 8192 bits
Accepted TLSv1.2 256 bits DHE-RSA-AES256-CCM DHE 8192 bits
-Accepted TLSv1.2 128 bits ECDHE-RSA-AES128-GCM-SHA256 Curve P-521 DHE 521
+Accepted TLSv1.2 128 bits ECDHE-RSA-AES128-GCM-SHA256 Curve 448 DHE 448
Accepted TLSv1.2 128 bits DHE-RSA-AES128-GCM-SHA256 DHE 8192 bits
Accepted TLSv1.2 128 bits DHE-RSA-AES128-CCM DHE 8192 bits
-Accepted TLSv1.2 256 bits ECDHE-RSA-AES256-SHA Curve P-521 DHE 521
+Accepted TLSv1.2 256 bits ECDHE-RSA-AES256-SHA Curve 448 DHE 448
Accepted TLSv1.2 256 bits DHE-RSA-AES256-SHA DHE 8192 bits
-Accepted TLSv1.2 128 bits ECDHE-RSA-AES128-SHA Curve P-521 DHE 521
+Accepted TLSv1.2 128 bits ECDHE-RSA-AES128-SHA Curve 448 DHE 448
Accepted TLSv1.2 128 bits DHE-RSA-AES128-SHA DHE 8192 bits
Accepted TLSv1.2 256 bits AES256-GCM-SHA384
Accepted TLSv1.2 256 bits AES256-CCM
@@ -50,8 +50,10 @@
Server Key Exchange Group(s):
TLSv1.3 260 bits secp521r1 (NIST P-521)
+TLSv1.3 224 bits x448
TLSv1.3 192 bits ffdhe8192
TLSv1.2 260 bits secp521r1 (NIST P-521)
+TLSv1.2 224 bits x448
SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
Test #15 FAILED.
--- docker_test/expected_output/test_15.txt 2025-01-26 00:19:21.826672304 +0000
+++ /tmp/sslscan_test-results_Qf5TlhAjUz/test_15.txt 2025-01-26 00:27:03.063104961 +0000
@@ -6,8 +6,8 @@
SSL/TLS Protocols:
SSLv2 disabled
SSLv3 disabled
-TLSv1.0 enabled
-TLSv1.1 enabled
+TLSv1.0 disabled
+TLSv1.1 disabled
TLSv1.2 enabled
TLSv1.3 enabled
@@ -15,7 +15,7 @@
Server supports TLS Fallback SCSV
TLS renegotiation:
-Secure session renegotiation supported
+Session renegotiation not supported
TLS Compression:
Compression disabled
@@ -23,8 +23,6 @@
Heartbleed:
TLSv1.3 not vulnerable to heartbleed
TLSv1.2 not vulnerable to heartbleed
-TLSv1.1 not vulnerable to heartbleed
-TLSv1.0 not vulnerable to heartbleed
Supported Server Cipher(s):
Preferred TLSv1.3 128 bits TLS_AES_128_GCM_SHA256 Curve 25519 DHE 253
@@ -38,16 +36,13 @@
Accepted TLSv1.2 128 bits ECDHE-ECDSA-AES128-CCM Curve 25519 DHE 253
Accepted TLSv1.2 256 bits ECDHE-ECDSA-AES256-SHA Curve 25519 DHE 253
Accepted TLSv1.2 128 bits ECDHE-ECDSA-AES128-SHA Curve 25519 DHE 253
-Preferred TLSv1.1 256 bits ECDHE-ECDSA-AES256-SHA Curve 25519 DHE 253
-Accepted TLSv1.1 128 bits ECDHE-ECDSA-AES128-SHA Curve 25519 DHE 253
-Preferred TLSv1.0 256 bits ECDHE-ECDSA-AES256-SHA Curve 25519 DHE 253
-Accepted TLSv1.0 128 bits ECDHE-ECDSA-AES128-SHA Curve 25519 DHE 253
Server Key Exchange Group(s):
TLSv1.3 128 bits secp256r1 (NIST P-256)
TLSv1.3 192 bits secp384r1 (NIST P-384)
TLSv1.3 260 bits secp521r1 (NIST P-521)
TLSv1.3 128 bits x25519
+TLSv1.3 224 bits x448
TLSv1.3 112 bits ffdhe2048
TLSv1.3 128 bits ffdhe3072
TLSv1.3 150 bits ffdhe4096
@@ -57,6 +52,7 @@
TLSv1.2 192 bits secp384r1 (NIST P-384)
TLSv1.2 260 bits secp521r1 (NIST P-521)
TLSv1.2 128 bits x25519
+TLSv1.2 224 bits x448
SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
Test #16 passed.
Test #17 passed.
Test #18 FAILED.
--- docker_test/expected_output/test_18.txt 2025-01-26 00:19:21.826672304 +0000
+++ /tmp/sslscan_test-results_Qf5TlhAjUz/test_18.txt 2025-01-26 00:27:06.343105429 +0000
@@ -33,6 +33,7 @@
TLSv1.2 192 bits secp384r1 (NIST P-384)
TLSv1.2 260 bits secp521r1 (NIST P-521)
TLSv1.2 128 bits x25519
+TLSv1.2 224 bits x448
SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
!! SOME TESTS FAILED !!
Generally, the Docker tests will spin up containers for your newly- built sslscan to run against. The output will be diff'ed against the expected output, and any variation will cause a failure. When new functionality is added, the expected results may change, and hence, the tests must be updated. Of course, failures can also be the result of new bugs introduced by the code update. In this case, since you're only removing supposedly dead code from the code base, the expected results should all be the same.
Test #13 failed (among other reasons) because TLS v1.0 and v1.1 are enabled on the test endpoint, but your sslscan build thinks they're disabled. Also, strangely, your sslscan build is detecting that x448 is enabled in TLS v1.2 and v1.3, even though the master branch does not.
My point was this was against the HEAD of the master branch without any local changes at all. My expectation would be there are no failed tests, but when I run it, the output I'm seeing is what I've put in this issue. Any pointers on where to look at this? Am I the only one seeing this?
For reference, I'm using an Ubuntu 22.04 aarch64 VM running on my Mac. I don't think that should make a difference, but I thought I would mention.
I just cloned the master branch into a fresh directory, built it with
make -f Makefile static, then ran ./docker_test.sh and got all
passing tests.
Just so we make extra sure, can you try the same? (i.e.: clone into a fresh directory just to make sure the master HEAD is being built & tested.)
For reference, I'm using an Ubuntu 22.04 aarch64 VM running on my Mac.
This would be my second suspicion. When you run docker_test.sh on a
new machine, it builds a Docker test image named "sslscan-test" with a
couple versions of OpenSSL and GnuTLS; the server executables in this
image are what the locally-built sslscan is tested against. I've only
built this image on x86_64 machines, but perhaps the resulting image is
different on aarch64! Hence, the problem may be with the test image
itself.
What git branch of yours is resulting in these failures? I'll try running tests on your branch with my x86_64 test image and see what happens.
I spun up an aarch64 machine in AWS and found that some of the tests fail. So this isn't a problem on your end. Interestingly, the tests fail for different reasons in AWS than they do for you!
Until this is fixed, I suppose I'll simply run the Docker tests on your PRs as they come in. Please ping me, and I'll do my best to be quick about it!