[BUG] Unknown: [ddappsec] Error receiving reply for command request_init: dd_network in Unknown on line 0

[joelharkes]

Bug description

Running in Azure AKS (using the helm chart, latest version) the following warnings are popping up. Locally i cannot reproduce in docker-compose.

NOTICE: PHP message: PHP Warning:  Unknown: [ddappsec] Error receiving reply for command request_init: dd_network in Unknown on line 0

PHP version

PHP 7.4.28 (cli) (built: Feb 28 2022 07:33:39) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Xdebug v3.0.4, Copyright (c) 2002-2021, by Derick Rethans
    with Zend OPcache v7.4.28, Copyright (c), by Zend Technologies

Tracer version

0.76.1

Installed extensions

[PHP Modules]
bcmath
bz2
calendar
Core
ctype
curl
date
dba
dom
exif
FFI
fileinfo
filter
ftp
gd
gettext
gmp
hash
iconv
intl
json
ldap
libxml
mbstring
mysqli
mysqlnd
odbc
openssl
pcntl
pcre
PDO
pdo_dblib
pdo_mysql
PDO_ODBC
pdo_pgsql
pdo_sqlite
pdo_sqlsrv
pgsql
Phar
phpdbg_webhelper
posix
pspell
readline
Reflection
session
shmop
SimpleXML
soap
sockets
sodium
SPL
sqlite3
sqlsrv
standard
sysvmsg
sysvsem
sysvshm
tidy
tokenizer
xdebug
xml
xmlreader
xmlrpc
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Xdebug
Zend OPcache

OS info

NAME="Alpine Linux" VERSION_ID=3.15.5 PRETTY_NAME="Alpine Linux v3.15"

Diagnostics and configuration

Output of phpinfo() (ddtrace >= 0.47.0)

cannot access this info as so easily, if necessary i can add it.

Upgrading info

[joelharkes]

where does this error come from?

[Anilm3]

Hey @joelharkes

Thanks for the bug report. This error is caused by the AppSec extension (https://github.com/DataDog/dd-appsec-php) and unfortunately can happen due to a large host of reasons. The best way for us to investigate further is to get some logs from you if possible. To enable debug logs on AppSec you'll need to set the following ini settings in the tracer configuration file:

datadog.appsec.log_level=‘debug’
datadog.appsec.helper_extra_args=‘--log_level=debug’
datadog.appsec.helper_log_file=‘/tmp/helper.log’
datadog.appsec.log_file=’/tmp/extension.log’

More details can be found in this troubleshooting doc. These logs could potentially contain PII so feel free to send them to datadog support, although please do mention that we requested these.

Alternatively if you enabled the AppSec extension by mistake, you can disable it by changing the datadog.appsec.enabled ini setting to Off in the tracer configuration file.

[joelharkes]

Ill try to see if i can get some more information.

[krakjoe]

Hi @joelharkes: bump for some more information :)

[krakjoe]

If this problem persists, please open a new bug report in the appsec extension repository for best results.

Closing this here for the moment.

[HadrienKulik]

Hi, we are having the same issue

PHP 8.0.19
   with ddtrace v0.80.0
   with ddappsec v0.4.3

This error is popping up a bit randomly (a refresh usually makes it disappear). Unknown: [ddappsec] Error receiving reply for command request_init: dd_network in Unknown on line 0

We have activated the debug logs as suggested above.

extension.log does not show much

[2022-10-19T10:29:48.319Z][7][debug] Appsec disabled at ddappsec.c:231:zm_activate_ddappsec
[2022-10-19T10:29:53.476Z][484][debug] Appsec disabled at ddappsec.c:231:zm_activate_ddappsec
[2022-10-19T10:29:53.839Z][484][debug] Shutting down the file logging at logging.c:464:dd_log_shutdown
[2022-10-19T10:29:53.890Z][512][debug] Appsec disabled at ddappsec.c:231:zm_activate_ddappsec
[2022-10-19T10:29:54.259Z][512][debug] Shutting down the file logging at logging.c:464:dd_log_shutdown
[2022-10-19T10:29:54.432Z][7][debug] Shutting down the file logging at logging.c:464:dd_log_shutdown
[2022-10-19T11:29:53.965Z][31][debug] Shutting down the file logging at logging.c:464:dd_log_shutdown

helper.log grew to 2+Go in about an hour of logging. I tried to grep for dd_network but can't find any mention of it. And a grep of warn yield lines such as:

[2022-10-19 13:34:41.182][warning][1641] Failed to handle message: Message body too large: 1952990169
[2022-10-19 13:34:41.324][warning][1641] Failed to handle message: Message body too large: 2019897135
[2022-10-19 13:34:41.653][warning][1914] Failed to unpack message: str size overflow
[2022-10-19 13:34:42.656][warning][5755] Failed to unpack message: str size overflow
[2022-10-19 13:34:43.088][warning][5777] Failed to unpack message: str size overflow
...