CESNET
Netopeer2-server crash with 2.2.28 with TLS configuration
Hi Michal, we have observed a netopeer2-server crash during tls call home, we wanted to know if this issue is ever seen or encountered, the backtrace points to strcmp_avx on certificate verification in libnetconf tls calls, Attached netopeer2 core and tls_certs used to configure the server.
BT:
warning: Unexpected size of section `.reg-xstate/379' in core file.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `netopeer2-server -d -v 2 -t 20 -x /usr/local/bin/scripts/mount-schema.xml'.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Unexpected size of section `.reg-xstate/379' in core file.
#0 0x00007febfda0080b in __strcmp_avx2 () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7febf67a1700 (LWP 379))]
Missing separate debuginfos, use: yum debuginfo-install netopeer2-gcc-x86-64-rocky8.10-release-prod-0.0.4_main-2.2.28.x86_64
(gdb) bt
#0 0x00007febfda0080b in _**_strcmp_avx2 ()** from /lib64/libc.so.6
#1 0x00007febfe860112 in **nc_server_tls_ks_ref_get_cert_key** () from /usr/local/lib64/libnetconf2.so.4
#2 0x00007febfe861cdc in nc_server_tls_load_server_cert_key () from /usr/local/lib64/libnetconf2.so.4
#3 0x00007febfe86274c in nc_accept_tls_session () from /usr/local/lib64/libnetconf2.so.4
#4 0x00007febfe841359 in nc_connect_ch_endpt () from /usr/local/lib64/libnetconf2.so.4
#5 0x00007febfe841af2 in nc_ch_client_thread () from /usr/local/lib64/libnetconf2.so.4
#6 0x00007febfe3721ca in start_thread () from /lib64/libpthread.so.0
#7 0x00007febfd98f8d3 in clone () from /lib64/libc.so.6
(gdb) quit
tls_certs.zip core.netopeer2-serve.0.be4dc4a67cf843b685cb16e6f59b8d15.944209.zip
Please include your YANG configuration of ietf-netconf-server (output of sysrepocfg -X -m ietf-netconf-server).
Hi michal, we dont have the same process running, but this is the configuration that is updated for netconf server,
sysrepocfg -X -m ietf-netconf-server
<netconf-server xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-server">
<listen>
<endpoints>
<endpoint>
<name>default-ssh</name>
<ssh>
<tcp-server-parameters>
<local-address>0.0.0.0</local-address>
</tcp-server-parameters>
<ssh-server-parameters>
<server-identity>
<host-key>
<name>default-key</name>
<public-key>
<central-keystore-reference>genkey</central-keystore-reference>
</public-key>
</host-key>
<host-key>
<name>default-key1</name>
<public-key>
<central-keystore-reference>ecdsakey</central-keystore-reference>
</public-key>
</host-key>
</server-identity>
<client-authentication>
<users>
<user>
<name>root</name>
<public-keys>
<use-system-keys xmlns="urn:cesnet:libnetconf2-netconf-server"/>
</public-keys>
</user>
</users>
</client-authentication>
</ssh-server-parameters>
</ssh>
</endpoint>
<endpoint>
<name>default-tls</name>
<tls>
<tcp-server-parameters>
<local-address>0.0.0.0</local-address>
</tcp-server-parameters>
<tls-server-parameters>
<server-identity>
<certificate>
<central-keystore-reference>
<asymmetric-key>serverkey</asymmetric-key>
<certificate>servercert</certificate>
</central-keystore-reference>
</certificate>
</server-identity>
<client-authentication>
<ca-certs>
<central-truststore-reference>cacerts</central-truststore-reference>
</ca-certs>
<ee-certs>
<central-truststore-reference>eecerts</central-truststore-reference>
</ee-certs>
</client-authentication>
</tls-server-parameters>
<netconf-server-parameters>
<client-identity-mappings>
<cert-to-name>
<id>10</id>
<fingerprint>02:cc:b3:ce:55:69:11:f5:ba:83:15:32:00:fb:c3:22:e3:2a:16:3e:41</fingerprint>
<map-type xmlns:x509c2n="urn:ietf:params:xml:ns:yang:ietf-x509-cert-to-name">x509c2n:specified</map-type>
<name>netconf</name>
</cert-to-name>
</client-identity-mappings>
</netconf-server-parameters>
</tls>
</endpoint>
</endpoints>
</listen>
<call-home>
<netconf-client>
<name>default-client-tls</name>
<endpoints>
<endpoint>
<name>default-tls</name>
<tls>
<tcp-client-parameters>
<remote-address>127.0.0.1</remote-address>
</tcp-client-parameters>
<tls-server-parameters>
<server-identity>
<certificate>
<central-keystore-reference>
<asymmetric-key>serverkey</asymmetric-key>
<certificate>servercert</certificate>
</central-keystore-reference>
</certificate>
</server-identity>
<client-authentication>
<ca-certs>
<central-truststore-reference>cacerts</central-truststore-reference>
</ca-certs>
<ee-certs>
<central-truststore-reference>eecerts</central-truststore-reference>
</ee-certs>
</client-authentication>
</tls-server-parameters>
<netconf-server-parameters>
<client-identity-mappings>
<cert-to-name>
<id>10</id>
<fingerprint>02:cc:b3:ce:55:69:11:f5:ba:83:15:32:00:fb:c3:22:e3:2a:16:3e:41</fingerprint>
<map-type xmlns:x509c2n="urn:ietf:params:xml:ns:yang:ietf-x509-cert-to-name">x509c2n:specified</map-type>
<name>netconf</name>
</cert-to-name>
</client-identity-mappings>
</netconf-server-parameters>
</tls>
</endpoint>
</endpoints>
<connection-type>
<persistent/>
</connection-type>
</netconf-client>
</call-home>
</netconf-server>
I have tested this configuration and it seems to be almost exactly the example configuration provided by netopeer2. It worked without any issues for me but I have used the latest versions of all the libraries so there may have been a problem fixed. My suggestion is to update to the latest release and try again. Or you can wait a bit until I make a new release, should be today or later this week.
Hi Michal,
We upgraded to the following versions of libraries: (libyang): upgrade to version v3.7.8 (libnetconf2): upgrade to version v3.5.5 (sysrepo): upgrade to version v3.3.10 (netopeer2): upgrade to version v2.2.35
But on a cyclic test with multiple iterations of config push, we got this crash once. Can you please suggest?
[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `netopeer2-server -d -v 2 -t 20 -x /usr/local/bin/scripts/mount-schema.xml'. Program terminated with signal SIGSEGV, Segmentation fault.
warning: Unexpected size of section `.reg-xstate/604' in core file. #0 0x00007fef48d2980b in __strcmp_avx2 () from /lib64/libc.so.6 [Current thread is 1 (Thread 0x7fef38dbe700 (LWP 604))] Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-251.el8_10.4.x86_64 keyutils-libs-1.5.10-9.el8.x86_64 krb5-libs-1.18.2-29.el8_10.x86_64 libatomic-8.5.0-22.el8_10.x86_64 libblkid-2.32.1-46.el8.x86_64 libcap-2.48-6.el8_9.x86_64 libcom_err-1.45.6-5.el8.x86_64 libcurl-7.61.1-34.el8_10.2.x86_64 libmount-2.32.1-46.el8.x86_64 libnghttp2-1.33.0-6.el8_10.1.x86_64 libselinux-2.9-8.el8.x86_64 libuuid-2.32.1-46.el8.x86_64 libxcrypt-4.1.1-6.el8.x86_64 openssl-libs-1.1.1k-12.el8_9.x86_64 systemd-libs-239-82.el8_10.1.x86_64 zlib-1.2.11-26.el8.x86_64 (gdb) bt #0 0x00007fef48d2980b in __strcmp_avx2 () from /lib64/libc.so.6 #1 0x00007fef49b2e281 in ?? () #2 0x00000000000002b1 in ?? () #3 0x00007fef10000bb0 in ?? () #4 0x0000000000000000 in ?? () (gdb) thread apply all bt
Seems like a bug but without a reproducible test case or at least a full stack trace I cannot help you.
There are no file lines but it seems like a very basic issue that would be encountered by many users if it were common. Which makes me think it is specific to your system, perhaps something related to the update, issues tend to occur if you keep all the old auxiliary files of the projects. So my suggestion would be to fully uninstall mainly sysrepo (by running make sr_clean in its build dir) and then try to install netopeer2 again.
Hi Michal,
we usually download source and doing a fresh build every time, we configure Build with cmake and package it to rpm, this rpm is freshly installed on the system ( no previous installation ) and then configured for TLS, I have uploaded configuration, and certificates used.
netconf server configuration
<netconf-server xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-server">
<listen>
<idle-timeout>0</idle-timeout>
<endpoints>
<endpoint>
<name>default-ssh</name>
<ssh>
<tcp-server-parameters>
<local-address>0.0.0.0</local-address>
</tcp-server-parameters>
<ssh-server-parameters>
<server-identity>
<host-key>
<name>default-key</name>
<public-key>
<central-keystore-reference>genkey</central-keystore-reference>
</public-key>
</host-key>
<host-key>
<name>default-key1</name>
<public-key>
<central-keystore-reference>ecdsakey</central-keystore-reference>
</public-key>
</host-key>
</server-identity>
<client-authentication>
<users>
<user>
<name>root</name>
<public-keys>
<use-system-keys xmlns="urn:cesnet:libnetconf2-netconf-server"/>
</public-keys>
</user>
</users>
</client-authentication>
</ssh-server-parameters>
</ssh>
</endpoint>
<endpoint>
<name>default-tls</name>
<tls>
<tcp-server-parameters>
<local-address>0.0.0.0</local-address>
</tcp-server-parameters>
<tls-server-parameters>
<server-identity>
<certificate>
<central-keystore-reference>
<asymmetric-key>serverkey</asymmetric-key>
<certificate>servercert</certificate>
</central-keystore-reference>
</certificate>
</server-identity>
<client-authentication>
<ca-certs>
<central-truststore-reference>cacerts</central-truststore-reference>
</ca-certs>
<ee-certs>
<central-truststore-reference>eecerts</central-truststore-reference>
</ee-certs>
</client-authentication>
</tls-server-parameters>
<netconf-server-parameters>
<client-identity-mappings>
<cert-to-name>
<id>10</id>
<fingerprint>02:cc:b3:ce:55:69:11:f5:ba:83:15:32:00:fb:c3:22:e3:2a:16:3e:41</fingerprint>
<map-type xmlns:x509c2n="urn:ietf:params:xml:ns:yang:ietf-x509-cert-to-name">x509c2n:specified</map-type>
<name>netconf</name>
</cert-to-name>
</client-identity-mappings>
</netconf-server-parameters>
</tls>
</endpoint>
</endpoints>
</listen>
<call-home>
<netconf-client>
<name>default-client-tls</name>
<endpoints>
<endpoint>
<name>default-tls</name>
<tls>
<tcp-client-parameters>
<remote-address>127.0.0.1</remote-address>
</tcp-client-parameters>
<tls-server-parameters>
<server-identity>
<certificate>
<central-keystore-reference>
<asymmetric-key>serverkey</asymmetric-key>
<certificate>servercert</certificate>
</central-keystore-reference>
</certificate>
</server-identity>
<client-authentication>
<ca-certs>
<central-truststore-reference>cacerts</central-truststore-reference>
</ca-certs>
<ee-certs>
<central-truststore-reference>eecerts</central-truststore-reference>
</ee-certs>
</client-authentication>
</tls-server-parameters>
<netconf-server-parameters>
<client-identity-mappings>
<cert-to-name>
<id>10</id>
<fingerprint>02:cc:b3:ce:55:69:11:f5:ba:83:15:32:00:fb:c3:22:e3:2a:16:3e:41</fingerprint>
<map-type xmlns:x509c2n="urn:ietf:params:xml:ns:yang:ietf-x509-cert-to-name">x509c2n:specified</map-type>
<name>netconf</name>
</cert-to-name>
</client-identity-mappings>
</netconf-server-parameters>
</tls>
</endpoint>
</endpoints>
<connection-type>
<persistent/>
</connection-type>
</netconf-client>
</call-home>
</netconf-server>
We have patched post install section of netopeer to have 2 keys configured for connectivity ( ssh-rsa key pair and ssh-ecdsa key pair),
Please let me know if sharing these patch information helps,
All Commands that happen as part of make install is now called in postinstall scriptlet in netopeer rpm that is built.
Please let me if any additional information is required to help debug the issue
I have tested it with the provided data and certificates, worked just fine and I established a TLS Call Home session. So yes, you can provide the exact patches you are using but at this point I am pretty sure some changes of yours are causing the problems.
001_cpack_changes_devel_prod.patch Hi Michal, Providing you patch, you can ignore, +include(CreateArtifact) +create_artifact_package(create_prod package_release ${OPEN_SOURCE_VERSION})
this includes CPack mk rules, i am also sharing the rpm built using this patch Most of changes are for supporting post install of rpm (Please check install.sh in patch ) [srikanthsubbaramu@lpt-fhyz314 Downloads]$ rpm -qp --scripts netopeer2-gcc-x86-64-rocky8.10-release-prod-0.7.0-feature-ran-2052-main-x86_64.rpm preinstall scriptlet (using /bin/sh): #!/bin/bash postinstall scriptlet (using /bin/sh):
#!/bin/bash echo "Executing the post-install scriptlet" sh /usr/local/share/netopeer2/scripts/install.sh
netopeer2-gcc-x86-64-rocky8.10-release-prod-0.7.0-feature-ran-2052-main-x86_64.rpm.zip
also please note this observation, this issue occurs 1 in 50 iterations of testing
Hi Michal, I think i know what could be problem, we are generating this xmls dynamically after downloading certs, is there a possibility of hitting this stack trace if any of field is loaded with empty value in keystore
<asymmetric-keys>
<asymmetric-key>
<name>serverkey</name>
<public-key-format xmlns:ct="urn:ietf:params:xml:ns:yang:ietf-crypto-types">ct:subject-public-key-info-format</public-key-format>
<public-key>MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAx1VDJjKI3XQYmooKH46cHEtFx5yokHm6uM5yo5/nTuxrGeARxNqsPhSFYp4BKsJcwQErAYq9sFl2ByccrYqzmr9AnsPTVHZRrdRVasLgoUIpQSHBLePogdJZ925pSOUGH+T8/7+89Gsl1ezNthrSf+PV/7N5xkgQXpcCMBjOF6a/yPy0XYFOZ0OE7ebFlffIWUcUNElkxIFY5ejeeN8+qluEVWYJrC64xEoPK18XnzCZVCZpMLFOI1nCQx5d06cCaHuzTMt5YYKhkCeAE0tJIRaFcR3C6Yd+HRz/ytWG+Jg30TqE3A1HoI1CrGxi1CcemTFGoyw4pzUPl9LeurXmwQIDAQAB</public-key>
<private-key-format xmlns:ct="urn:ietf:params:xml:ns:yang:ietf-crypto-types">ct:rsa-private-key-format</private-key-format>
<cleartext-private-key>MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCo1mfA1qmW1YUSgQ1iPTicYPC7GH3L6ZhZxRwdb2XwbApGj/jffkNhSpYBAFqnqjeEaqzX/U1eBG6LFUvRry3eLvr7tlUL+w3DZgLcBzlSPRTTn8w+cCejE0nRhL1XeNnwM52CqlHhqvuUadQQGAANOisH95IPb6Qf7JVAoBdO8VUVLMQ9LoUFp/qXXRPjKgjbYmqf27FofmpK9zCBNfj2m7nDO45BZYN1+xNP4IogcVqdN0wbmt3uPCseK2Kcu3UPBP8OtsDBCWOt+kA7AyHAYpVJGu6hlMK/oAWVIfxM0dyPzyg1FhIRE2lMUbhaC0cvAfsO0qYQLMy+ruCGDebpAgMBAAECggEABxf3BP2f43P2IthkOQ/sbHmQM7QsOOCII6Fp9HylMkw/xEYxRSaXayOImOMsa+X1bi1TNMHyObSC9nn/FQDAsxiTN/cprJawNdj33sm46VEiql+I48iviaT6UYC7ucycj4CDqiVAynP2HP6zR7fSbLvYaf3HV7mvkh3NCYmQYSMmBpIPBzUpwnW78xo0X+jMVlRmuDSH8z3xMoQHBMZgpxKtY2PYCpYeKlLSuQiBlXJCF6bMHRsaO+BRlRA+AsGix8TGOGyHW+Wq1wHAiNbfzi/UwIbd57dxc9ac7QkN4jLQ26wCskSrbayZQvSt0KVgtVaVTlSJw2rdcI1hKkiNWQKBgQC6al7iDAZtPa/otDYqLFmBB+f4rKqgcphHVaDFPtjofnn0YnmTXLRwV/ooSGtF9ULpZMP/Se1wOO5FWnboPhKMD97GAGGundc+ylHgHz/tjGzwwpZ+YORqfLxu8OyxYoE0HKOilrcmIruL0gD+u1zbUxtnEJZnpXCDwLOgpeFxLQKBgQDn3E3lyw4QSQ/hF4cYYDHNrt8VVuGnOa+dINPtw76ZbCd+D82zc0v9dcVRf46u87zrTUlYQ4CVjIfqkELP55hDvy3Y/zN7P/fr1Zw9GIRXS2A9zdD2FDcvj0KJDbvgw3sAcxbZCR2i9ko0QzwPW1nsO+oNGRKJcy8emrQa805KLQKBgHQ1kXbLVkpNdVbU2RtLUHSekB62zRt+tK1rlPDBcAjnp3EQ3odd+GI8hgcMtksDTTYgCgsgc/NMmkUD0zKOV5OW5SJ75GktnpxXFdlowbp9mwAv3g9kqaA0qGdkq7kdFjx9Sgk2eXA9oLrWLKaf+JAFbATBE3IDcXPA8nnITxT1AoGALETq9qITeFaK2p5kY+oR+ESYQXnKMeSYvDaFYFNMc/yrea1IyCeObcFrwEjLlGnjO0YRZ/HTfjpLxSRwLUP51Y2OEm1/hdvL2VJ6t0uUERrKMGK4sBNiCgmfWY2uvpZ0SLywsxXDe9bsihgAQqpde/ZglMmhuW6to3lERBUKcK0CgYBSIysln1CxxWR+DcGaA3xtCogys+AXV+yO46DjOeim5w+YTjHHcSYVCvib5Edjr89cggQAuTfG/EdNHwBHjvTKoiunIFMkZXe7a8LqTDDCHsh+RPoLEUF/yQHOChAhq/M3Z+KOqW3tAsaxGOJAcW3COG5teMC//MJKyzaj0Fe8+A==</cleartext-private-key>
In that case, i think i can may be dump the content of file on to logs, so it would help troubleshoot.
Does the stacktrace
#0 0x00007fef48d2980b in __strcmp_avx2 () from /lib64/libc.so.6
#1 0x00007fef49b2e281 in nc_server_tls_load_server_cert_key () from /usr/local/lib64/libnetconf2.so.4
#2 0x00007fef49b2e4fc in nc_accept_tls_session () from /usr/local/lib64/libnetconf2.so.4
does this indicate that serverkey config may have not been proper?
Thanks, Srikanth
Yes, something like this could be a problem. What does "generating this xmls dynamically after downloading certs" means exactly? How do you configure sysrepo to use these XMLs?
Basically i am using below script, which reads the certs provided in tgz, and generates xmls for adding to server, The certs directory contains below [root@server0 tmp]# ls tls_certs ca-cert.pem ca-cert.srl ca-key.pem client-key.pem client.crt client.csr publickey.pem server-key.pem server.crt server.csr
The attached scripts generate the required config generating keys and update variables in xml, which are sent to server for config, in random runs, we see crash, so trying to figure out which of the field can cause the issue
I purposefully, did empty private key in keystore.xml, empty public key , But netopeer2-server printerd error in logs indicating key is empty, so trying to figureout what could cause the issue
Yes, the data are then always valid so I do not understand how an issue can occur. Unless I can reproduce it, I cannot help you it seems.
Hi Michal,
We tried to replicate issue collecting data, with netopeer2-cli using same data, but we were unable to reproduce the issue locally with netopeer2-cli on same vm, But when there is a callhome done to a other application, which listens to connect with tls, the crash is seen, we have been able to collect configuration uploaded to netconf server, and also backtrace
keystore
<keystore xmlns="urn:ietf:params:xml:ns:yang:ietf-keystore">
<asymmetric-keys>
<asymmetric-key>
<name>serverkey</name>
<public-key-format xmlns:ct="urn:ietf:params:xml:ns:yang:ietf-crypto-types">ct:subject-public-key-info-format</public-key-format>
<public-key>MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAx1VDJjKI3XQYmooKH46cHEtFx5yokHm6uM5yo5/nTuxrGeARxNqsPhSFYp4BKsJcwQErAYq9sFl2ByccrYqzmr9AnsPTVHZRrdRVasLgoUIpQSHBLePogdJZ925pSOUGH+T8/7+89Gsl1ezNthrSf+PV/7N5xkgQXpcCMBjOF6a/yPy0XYFOZ0OE7ebFlffIWUcUNElkxIFY5ejeeN8+qluEVWYJrC64xEoPK18XnzCZVCZpMLFOI1nCQx5d06cCaHuzTMt5YYKhkCeAE0tJIRaFcR3C6Yd+HRz/ytWG+Jg30TqE3A1HoI1CrGxi1CcemTFGoyw4pzUPl9LeurXmwQIDAQAB</public-key>
<private-key-format xmlns:ct="urn:ietf:params:xml:ns:yang:ietf-crypto-types">ct:rsa-private-key-format</private-key-format>
<cleartext-private-key>MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCo1mfA1qmW1YUSgQ1iPTicYPC7GH3L6ZhZxRwdb2XwbApGj/jffkNhSpYBAFqnqjeEaqzX/U1eBG6LFUvRry3eLvr7tlUL+w3DZgLcBzlSPRTTn8w+cCejE0nRhL1XeNnwM52CqlHhqvuUadQQGAANOisH95IPb6Qf7JVAoBdO8VUVLMQ9LoUFp/qXXRPjKgjbYmqf27FofmpK9zCBNfj2m7nDO45BZYN1+xNP4IogcVqdN0wbmt3uPCseK2Kcu3UPBP8OtsDBCWOt+kA7AyHAYpVJGu6hlMK/oAWVIfxM0dyPzyg1FhIRE2lMUbhaC0cvAfsO0qYQLMy+ruCGDebpAgMBAAECggEABxf3BP2f43P2IthkOQ/sbHmQM7QsOOCII6Fp9HylMkw/xEYxRSaXayOImOMsa+X1bi1TNMHyObSC9nn/FQDAsxiTN/cprJawNdj33sm46VEiql+I48iviaT6UYC7ucycj4CDqiVAynP2HP6zR7fSbLvYaf3HV7mvkh3NCYmQYSMmBpIPBzUpwnW78xo0X+jMVlRmuDSH8z3xMoQHBMZgpxKtY2PYCpYeKlLSuQiBlXJCF6bMHRsaO+BRlRA+AsGix8TGOGyHW+Wq1wHAiNbfzi/UwIbd57dxc9ac7QkN4jLQ26wCskSrbayZQvSt0KVgtVaVTlSJw2rdcI1hKkiNWQKBgQC6al7iDAZtPa/otDYqLFmBB+f4rKqgcphHVaDFPtjofnn0YnmTXLRwV/ooSGtF9ULpZMP/Se1wOO5FWnboPhKMD97GAGGundc+ylHgHz/tjGzwwpZ+YORqfLxu8OyxYoE0HKOilrcmIruL0gD+u1zbUxtnEJZnpXCDwLOgpeFxLQKBgQDn3E3lyw4QSQ/hF4cYYDHNrt8VVuGnOa+dINPtw76ZbCd+D82zc0v9dcVRf46u87zrTUlYQ4CVjIfqkELP55hDvy3Y/zN7P/fr1Zw9GIRXS2A9zdD2FDcvj0KJDbvgw3sAcxbZCR2i9ko0QzwPW1nsO+oNGRKJcy8emrQa805KLQKBgHQ1kXbLVkpNdVbU2RtLUHSekB62zRt+tK1rlPDBcAjnp3EQ3odd+GI8hgcMtksDTTYgCgsgc/NMmkUD0zKOV5OW5SJ75GktnpxXFdlowbp9mwAv3g9kqaA0qGdkq7kdFjx9Sgk2eXA9oLrWLKaf+JAFbATBE3IDcXPA8nnITxT1AoGALETq9qITeFaK2p5kY+oR+ESYQXnKMeSYvDaFYFNMc/yrea1IyCeObcFrwEjLlGnjO0YRZ/HTfjpLxSRwLUP51Y2OEm1/hdvL2VJ6t0uUERrKMGK4sBNiCgmfWY2uvpZ0SLywsxXDe9bsihgAQqpde/ZglMmhuW6to3lERBUKcK0CgYBSIysln1CxxWR+DcGaA3xtCogys+AXV+yO46DjOeim5w+YTjHHcSYVCvib5Edjr89cggQAuTfG/EdNHwBHjvTKoiunIFMkZXe7a8LqTDDCHsh+RPoLEUF/yQHOChAhq/M3Z+KOqW3tAsaxGOJAcW3COG5teMC//MJKyzaj0Fe8+A==</cleartext-private-key>
<certificates>
<certificate>
<name>servercert</name>
<cert-data>MIIDoDCCAoigAwIBAgIUeMRsLi9HI+OUoU1T1roxfHUwKXgwDQYJKoZIhvcNAQELBQAwZzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVN0YXRlMQ0wCwYDVQQHDARDaXR5MRUwEwYDVQQKDAxPcmdhbml6YXRpb24xEzARBgNVBAsMCkRlcGFydG1lbnQxDTALBgNVBAMMBE15Q0EwHhcNMjQxMDEwMDYwMzQ0WhcNMjUxMDEwMDYwMzQ0WjAcMRowGAYDVQQDDBFuZXRvcGVlci1zZXJ2ZXItMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKjWZ8DWqZbVhRKBDWI9OJxg8LsYfcvpmFnFHB1vZfBsCkaP+N9+Q2FKlgEAWqeqN4RqrNf9TV4EbosVS9GvLd4u+vu2VQv7DcNmAtwHOVI9FNOfzD5wJ6MTSdGEvVd42fAznYKqUeGq+5Rp1BAYAA06Kwf3kg9vpB/slUCgF07xVRUsxD0uhQWn+pddE+MqCNtiap/bsWh+akr3MIE1+PabucM7jkFlg3X7E0/giiBxWp03TBua3e48Kx4rYpy7dQ8E/w62wMEJY636QDsDIcBilUka7qGUwr+gBZUh/EzR3I/PKDUWEhETaUxRuFoLRy8B+w7SphAszL6u4IYN5ukCAwEAAaOBjjCBizAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwJAYDVR0RBB0wG4cEClkBAocEfwAAAYIIbmNzZXJ2ZXKCA25jMTAdBgNVHQ4EFgQUrJ/RHc2UG3Bf9y6G4fjDNWx+dK4wHwYDVR0jBBgwFoAUUCcpskwzX4vUuuB8PZq9sQVGryMwDQYJKoZIhvcNAQELBQADggEBAAosc49peeWyhRc5anbAsm63xzVIolv2jtxcfoXSw95GBbZGjxXuonilLa5zWPD+EykMZx8T8n/sUWwk2jQaQXxL0V/QK8hQlv6QsqvGPo/FwcYJdhuePn+pOuGpx+2pSM3lWd99lgM6aWWQ/c6TVdLiDZ/smS2snnTayQwahpvECv/ncWHCfQJH1T4ao284mYKdGvDgnnbk7BG1CjRrRPoyWjkB6XoQaRYlzVwUWH9tz36eVUpr7udSqIfCJQzlxkzqEO+LTT4jDhO0WVirfMWJmBttURm+E4vxAh642r+7JWYeMgAQMLbtJQtd8Fw8fySSIIpjfS3o6FfJtRleGgc=</cert-data>
</certificate>
</certificates>
</asymmetric-key>
</asymmetric-keys>
</keystore>
truststore
<truststore xmlns="urn:ietf:params:xml:ns:yang:ietf-truststore">
<certificate-bags>
<certificate-bag>
<name>eecerts</name>
<certificate>
<name>eecert</name>
<cert-data>MIIDVzCCAj8CFENwl1uwt91UTxk9mZh3Us0HDzwqMA0GCSqGSIb3DQEBCwUAMGcxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIDAVTdGF0ZTENMAsGA1UEBwwEQ2l0eTEVMBMGA1UECgwMT3JnYW5pemF0aW9uMRMwEQYDVQQLDApEZXBhcnRtZW50MQ0wCwYDVQQDDARNeUNBMB4XDTI0MTAwOTEzMzMzM1oXDTI1MTAwOTEzMzMzM1owaTELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVN0YXRlMQ0wCwYDVQQHDARDaXR5MRUwEwYDVQQKDAxPcmdhbml6YXRpb24xEzARBgNVBAsMCkRlcGFydG1lbnQxDzANBgNVBAMMBmNsaWVudDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAN66ciGTOrodr+kdwMMoGjpL/N4LQ4x9/c/bKyQTRxXyf8r/vfz0NupT0CM8FmJ8LsSAtuaA3O1wzC54uO9UNPyINQTdyRxf/mSzO5JCK6yExXkZsw8k7ghH2N+lFGA+KPsnZ506fqrBBAGhpkEj1PtgiK5ddDZ6bu/rExMAyMhMw+XtPBMbFTrOcv4YGyUVr/+9FPKdGMtijr/AxqIilwu8ApfuFiyMcpNrIpXxU0RnPqOuOHckfGJTPx6KJm1rehEgnmdvmscLFx7sjwWoNFe94bMm1KI4gZU534odBdLzFYmbJOOJxGIZCShw0IA25rcEzx2mDJn8FlUi3kuE66ECAwEAATANBgkqhkiG9w0BAQsFAAOCAQEANivpxrlGLMfabDd9NI9nbsy6qtk+6MIj1OThaM7ELwzgGfR0Qyjgbj/5QYR9zy53vK110jmvVvwqgDeffH3UOWwG7Y0PPqSwyG6kL9aTVSNUpKtUfubBF4Lm3HlOcC+nnktVLg4xBCM4RjPh+HAl/dXd/Vrlvj3D3KIyUb+n7dldpUdndGvMufgUBybeRlqT3FMlpqWl7ByTGZTm6rlWhUVHwMQ86FeBunLeC8PXB26hoKvzlCwXtEHxo44H5mI0lVaHBW+cpRQG+9bs6HLIml7d7w2R4EYTCuDsuisYfwlj5+P58dIToAvVXy/g1PwfyaQEX+LRTe6zgGaAto+lDA==</cert-data>
</certificate>
</certificate-bag>
<certificate-bag>
<name>cacerts</name>
<certificate>
<name>cacert</name>
<cert-data>MIIDrzCCApegAwIBAgIUazBN7AU3vSlUn1eu709ttgvdT4wwDQYJKoZIhvcNAQELBQAwZzELMAkGA1UEBhMCVVMxDjAMBgNVBAgMBVN0YXRlMQ0wCwYDVQQHDARDaXR5MRUwEwYDVQQKDAxPcmdhbml6YXRpb24xEzARBgNVBAsMCkRlcGFydG1lbnQxDTALBgNVBAMMBE15Q0EwHhcNMjQxMDA5MTMzMjQyWhcNMjUxMDA5MTMzMjQyWjBnMQswCQYDVQQGEwJVUzEOMAwGA1UECAwFU3RhdGUxDTALBgNVBAcMBENpdHkxFTATBgNVBAoMDE9yZ2FuaXphdGlvbjETMBEGA1UECwwKRGVwYXJ0bWVudDENMAsGA1UEAwwETXlDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL7PCujSdty8HsEqpOoTD2BPJQHVXxm4UKimdUgOcZywIqUrns5iE8Da2MfKz4PksfK7d+jkG1Q663cSwVIAPQrxoZqFiD1CoCrG0l9a+HdIZQXKDTNeCum3wgaJ4c0lXG0EjMdmskLhC12OKFaj86Ix75NcIBX4xxcB3A10heefnAkPZTqGTARVNYUW+YJvAFMZCspy8lTFCHfnrWATY3cA5zRQyphW0dv3k3maabS73dVazF2NytSbTh9Is6jNq2CKWeCsZP9HB5C1B5RaSwXVl5k3jicccCUNfMLfgcV7tKBdpNZAZCzh3ycVH4a898+WH+Qt1nPSWFmSnYcW/oECAwEAAaNTMFEwHQYDVR0OBBYEFFAnKbJMM1+L1LrgfD2avbEFRq8jMB8GA1UdIwQYMBaAFFAnKbJMM1+L1LrgfD2avbEFRq8jMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAD5o5w8XrXcNeQlDnfM1KO8IMEMeUo5L0SlDaW5SSB8vB5QoGhSZzgvrfAJbyRBuwFxJzYbDx7vwTbq5RCDEdoYJlEPkxylL09xbx1HedCXm+eICWyaCFSVuLVf7I/kLGhUn7/WteAf+BqoCJTCwfGL+M3aZmbNAMWRlGD8a+rJOw8EmCOEYC8EsdYkvgttrmqvwqXsGPo9sNVA8rAxoGERsnbmdwdSs8MQjB36WIjlx9Dzu9JOQdGqEcxOILCAL03StonnGE0q8fi/Vq+NU65rTijizsGf//H9jq4SSbB5B8vohCUOu7KjrytxeBisCK4VcKunuVFNS8F6Bc2kn/lk=</cert-data>
</certificate>
</certificate-bag>
</certificate-bags>
</truststore>
listen
<netconf-server xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-server">
<listen>
<endpoints>
<endpoint>
<name>default-tls</name>
<tls>
<tcp-server-parameters>
<local-address>0.0.0.0</local-address>
</tcp-server-parameters>
<tls-server-parameters>
<server-identity>
<certificate>
<central-keystore-reference>
<asymmetric-key>serverkey</asymmetric-key>
<certificate>servercert</certificate>
</central-keystore-reference>
</certificate>
</server-identity>
<client-authentication>
<ca-certs>
<central-truststore-reference>cacerts</central-truststore-reference>
</ca-certs>
<ee-certs>
<central-truststore-reference>eecerts</central-truststore-reference>
</ee-certs>
</client-authentication>
</tls-server-parameters>
<netconf-server-parameters>
<client-identity-mappings>
<cert-to-name>
<id>10</id>
<fingerprint>02:CC:B3:CE:55:69:11:F5:BA:83:15:32:00:FB:C3:22:E3:2A:16:3E:41</fingerprint>
<map-type xmlns:x509c2n="urn:ietf:params:xml:ns:yang:ietf-x509-cert-to-name">x509c2n:specified</map-type>
<name>netconf</name>
</cert-to-name>
</client-identity-mappings>
</netconf-server-parameters>
</tls>
</endpoint>
</endpoints>
</listen>
</netconf-server>
callhome
<netconf-server xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-server">
<call-home>
<netconf-client>
<name>default-client-tls</name>
<endpoints>
<endpoint>
<name>default-tls</name>
<tls>
<tcp-client-parameters>
<remote-address>10.3.8.52</remote-address>
</tcp-client-parameters>
<tls-server-parameters>
<server-identity>
<certificate>
<central-keystore-reference>
<asymmetric-key>serverkey</asymmetric-key>
<certificate>servercert</certificate>
</central-keystore-reference>
</certificate>
</server-identity>
<client-authentication>
<ca-certs>
<central-truststore-reference>cacerts</central-truststore-reference>
</ca-certs>
<ee-certs>
<central-truststore-reference>eecerts</central-truststore-reference>
</ee-certs>
</client-authentication>
</tls-server-parameters>
<netconf-server-parameters>
<client-identity-mappings>
<cert-to-name>
<id>10</id>
<fingerprint>02:CC:B3:CE:55:69:11:F5:BA:83:15:32:00:FB:C3:22:E3:2A:16:3E:41</fingerprint>
<map-type xmlns:x509c2n="urn:ietf:params:xml:ns:yang:ietf-x509-cert-to-name">x509c2n:specified</map-type>
<name>netconf</name>
</cert-to-name>
</client-identity-mappings>
</netconf-server-parameters>
</tls>
</endpoint>
</endpoints>
<connection-type>
<persistent/>
</connection-type>
</netconf-client>
</call-home>
</netconf-server>
nacm
<nacm xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-acm">
<groups>
<group>
<name>admin</name>
<user-name>netconf</user-name>
</group>
</groups>
<rule-list>
<name>all</name>
<group>admin</group>
<rule>
<name>all1</name>
<module-name>*</module-name>
<access-operations>*</access-operations>
<action>permit</action>
</rule>
</rule-list>
</nacm>
After config:
sysrepocfg -X -m ietf-netconf-server
<netconf-server xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-server">
<listen>
<endpoints>
<endpoint>
<name>default-ssh</name>
<ssh>
<tcp-server-parameters>
<local-address>0.0.0.0</local-address>
</tcp-server-parameters>
<ssh-server-parameters>
<server-identity>
<host-key>
<name>default-key</name>
<public-key>
<central-keystore-reference>genkey</central-keystore-reference>
</public-key>
</host-key>
<host-key>
<name>default-key1</name>
<public-key>
<central-keystore-reference>ecdsakey</central-keystore-reference>
</public-key>
</host-key>
</server-identity>
<client-authentication>
<users>
<user>
<name>root</name>
<public-keys>
<use-system-keys xmlns="urn:cesnet:libnetconf2-netconf-server"/>
</public-keys>
</user>
</users>
</client-authentication>
</ssh-server-parameters>
</ssh>
</endpoint>
<endpoint>
<name>default-tls</name>
<tls>
<tcp-server-parameters>
<local-address>0.0.0.0</local-address>
</tcp-server-parameters>
<tls-server-parameters>
<server-identity>
<certificate>
<central-keystore-reference>
<asymmetric-key>serverkey</asymmetric-key>
<certificate>servercert</certificate>
</central-keystore-reference>
</certificate>
</server-identity>
<client-authentication>
<ca-certs>
<central-truststore-reference>cacerts</central-truststore-reference>
</ca-certs>
<ee-certs>
<central-truststore-reference>eecerts</central-truststore-reference>
</ee-certs>
</client-authentication>
</tls-server-parameters>
<netconf-server-parameters>
<client-identity-mappings>
<cert-to-name>
<id>10</id>
<fingerprint>02:cc:b3:ce:55:69:11:f5:ba:83:15:32:00:fb:c3:22:e3:2a:16:3e:41</fingerprint>
<map-type xmlns:x509c2n="urn:ietf:params:xml:ns:yang:ietf-x509-cert-to-name">x509c2n:specified</map-type>
<name>netconf</name>
</cert-to-name>
</client-identity-mappings>
</netconf-server-parameters>
</tls>
</endpoint>
</endpoints>
</listen>
<call-home>
<netconf-client>
<name>default-client-tls</name>
<endpoints>
<endpoint>
<name>default-tls</name>
<tls>
<tcp-client-parameters>
<remote-address>10.3.8.52</remote-address>
</tcp-client-parameters>
<tls-server-parameters>
<server-identity>
<certificate>
<central-keystore-reference>
<asymmetric-key>serverkey</asymmetric-key>
<certificate>servercert</certificate>
</central-keystore-reference>
</certificate>
</server-identity>
<client-authentication>
<ca-certs>
<central-truststore-reference>cacerts</central-truststore-reference>
</ca-certs>
<ee-certs>
<central-truststore-reference>eecerts</central-truststore-reference>
</ee-certs>
</client-authentication>
</tls-server-parameters>
<netconf-server-parameters>
<client-identity-mappings>
<cert-to-name>
<id>10</id>
<fingerprint>02:cc:b3:ce:55:69:11:f5:ba:83:15:32:00:fb:c3:22:e3:2a:16:3e:41</fingerprint>
<map-type xmlns:x509c2n="urn:ietf:params:xml:ns:yang:ietf-x509-cert-to-name">x509c2n:specified</map-type>
<name>netconf</name>
</cert-to-name>
</client-identity-mappings>
</netconf-server-parameters>
</tls>
</endpoint>
</endpoints>
<connection-type>
<persistent/>
</connection-type>
</netconf-client>
</call-home>
</netconf-server>
So it makes me suspect, if there is a problem with data sent over from the client,
[INF]: LN: Call Home client "default-client-tls" endpoint "default-tls" connecting...
[INF]: LN: Trying to connect via IPv4 to 10.3.8.52:4335.
[INF]: LN: Successfully connected to 10.3.8.52:4335 over IPv4.
/usr/local/bin/scripts/netconf-server.sh: line 76: 640 Segmentation fault (core dumped) netopeer2-server -d -v 2 -t 20 -x $mountSchemaFile
[root@gnodeb-cucp-smo-9-cucp-cucp-smo-9-cm-agent-0 /]# gdb /usr/local/sbin/netopeer2-server /var/crash/core.\!usr\!local\!sbin\!netopeer2-server.640.gnodeb-cucp-smo-9-cucp-cucp-smo-9-cm-agent-0
GNU gdb (GDB) Rocky Linux 8.2-20.el8.0.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/netopeer2-server...(no debugging symbols found)...done.
warning: Can't open file (null) during file-backed mapping note processing
warning: core file may not match specified executable file.
[New LWP 825]
[New LWP 643]
[New LWP 646]
[New LWP 641]
[New LWP 640]
[New LWP 649]
[New LWP 647]
[New LWP 650]
[New LWP 655]
[New LWP 645]
[New LWP 648]
[New LWP 651]
[New LWP 653]
[New LWP 652]
[New LWP 654]
[New LWP 642]
[New LWP 644]
[New LWP 815]
[New LWP 816]
warning: Unexpected size of section `.reg-xstate/825' in core file.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `netopeer2-server -d -v 2 -t 20 -x /usr/local/bin/scripts/mount-schema.xml'.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Unexpected size of section `.reg-xstate/825' in core file.
#0 0x00007fc6775d380b in __strcmp_avx2 () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7fc65a7fc700 (LWP 825))]
Missing separate debuginfos, use: yum debuginfo-install netopeer2-gcc-x86-64-rocky8.10-release-prod-0.9.0_main-2.2.28.x86_64
(gdb) set pagination off
(gdb) bt full
#0 0x00007fc6775d380b in __strcmp_avx2 () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007fc6783ce4d1 in nc_server_tls_load_server_cert_key () from /usr/local/lib64/libnetconf2.so.4
No symbol table info available.
#2 0x00007fc6783ceab5 in nc_accept_tls_session () from /usr/local/lib64/libnetconf2.so.4
No symbol table info available.
#3 0x00007fc6783b2448
in nc_ch_client_thread () from /usr/local/lib64/libnetconf2.so.4
No symbol table info available.
#4 0x00007fc677f011ca in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007fc6775628d3 in clone () from /lib64/libc.so.6
No symbol table info available.
(gdb)
my suspect is there is some problem from data that is received from the app thats accepting callhome, can we instrument some debugging in netopeer2 to identify this problem? even if there was some problem in data, we should not crash but reject connection
The configuration you are configuring seems not to matter because it is completely different from the one from the sysrepocfg output. I am really not able to help you based on such chaotic reports.
netopeer2_server_crash_523.zip Hi Michal,
Issue recreated with debug enable and -c MSG logs, Please check if it helps have attached core, backtrace and console log for netopeer2-server Thanks, Srikanth
Thanks, please try using https://github.com/CESNET/libnetconf2/commit/956e73fc850b1305ca1e56e7b907cd24ac35982b.
HI Michal, we observed crash on 67th iteration,
Can you please check?
Hi, I put together some changes that should hopefully fix this issue. Please try using the latest devel branch of libnetconf2 and let us know, since we can't reproduce it. Also please let us know if you observe any noticeable differences in performance, thank you.
We were using versions Libyang v3.7.8 Libnetconf2 V3.5.5 sysrepo v3.3.10 netopeer2 2.2.35
do i need to move to devel branch in all repos? because of build dependency, i moved to libyang 3.12.2 , server init fialed, [ERR]: NP: Module "ietf-netconf-monitoring" not implemented in sysrepo. [ERR]: NP: Server init failed. [INF]: NP: Server terminated. [INF]: SR: Connection 99 destroyed.
Please suggest
Yes, you would need to move to devel branch of all the 4 projects. I have made some further improvements. Also my guess is that the sysrepo shm is corrupted, try cleaning it up by using make sr_clean in sysrepo's build/fresh install. If that doesn't work I could try and create a patch with the fix for libnetconf2, but not sure that would be ideal.
Hi ,
we have completed over 150+ iterations with devel branches of libyang, libnetconf2, sysrepo and netopeer2, we have not observed any crash,
can you please let us know, what specific performance differences we need to observe? does the change apply or contribute to only callhome case? for example: measure time taken for call home initiation and completeion?
Thanks, Srikanth
Hi, that is good to hear. If you are accepting call-home sessions in one thread and changing the configuration in another in each iteration, then each of these iterations should take a little longer now. It applies only to such Call Home scenarios.
For us , we do not have this scenario, the callhome client is the only client, that does the config update for everybody, But please let us know, when these will be release in official tag Thank you so much
The last release is still not particularly old so it will not be less than a month. However, if we decide we also want to release some other major features we are working on (in libyang, but we do all releases together), it may take longer.
Michal, do you have a view now on when the next release can be, including this fix?
I cannot really tell you any specific date, perhaps a few weeks up to a month or so.
Michal, we keep hitting this issue occasionally. We wanted to go with official release and not use the devel tagged fix. Please inform us if your next release plan is finalized.