trafficserver icon indicating copy to clipboard operation
trafficserver copied to clipboard
verified publisher icon apache

Memory leak in TS 8.0.6

Open jvgutierrez opened this issue 5 years ago • 5 comments

We are seeing what it looks like a memory leak in TS 8.0.6: image In the screenshot you can see cp1081 being affected by it, beginning on April 2nd at 09:28. From the logs I cannot see anything relevant at that time:

Apr 02 08:11:59 cp1081 traffic_ctl[42743]: NOTE: using command line path as RUNROOT
Apr 02 08:11:59 cp1081 traffic_ctl[42743]: NOTE: using command line path as RUNROOT
Apr 02 08:11:59 cp1081 traffic_manager[43724]: [Apr  2 08:11:59.769] traffic_manager NOTE: User has changed config file ssl_multicert.config
Apr 02 08:11:59 cp1081 systemd[1]: Reloaded trafficserver-tls.service.
Apr 02 08:33:59 cp1081 traffic_manager[43724]: [Apr  2 08:33:59.876] [ET_NET 64] ERROR: HTTP/2 connection error client_ip=REDACTED session_id=REDACTED stream_id=1 recv headers compression error
Apr 02 10:02:17 cp1081 traffic_manager[43724]: [Apr  2 10:02:17.702] [ET_NET 8] ERROR: HTTP/2 connection error client_ip=REDACTED session_id=REDACTED stream_id=0 priority 0 stream_id
Apr 02 13:50:29 cp1081 traffic_manager[43724]: [Apr  2 13:50:29.757] [ET_NET 63] ERROR: HTTP/2 connection error client_ip=REDACTED session_id=REDACTED stream_id=1 reset frame bad id stream not found

Configuration for cp1081:
vgutierrez@cp1081:~$ sudo -i cat /srv/trafficserver/tls/etc/remap.config
map wss://etherpad.wikimedia.org ws://10.64.16.24:3120
map / http://10.64.16.24:3120

vgutierrez@cp1081:~$ sudo -i cat /srv/trafficserver/tls/etc/ssl_multicert.config
dest_ip=* ssl_cert_name=acmecerts/unified/live/rsa-2048.chained.crt,acmecerts/unified/live/ec-prime256v1.chained.crt ssl_key_name=acmecerts/unified/live/rsa-2048.key,acmecerts/unified/live/ec-prime256v1.key ssl_ocsp_name=acmecerts/unified/live/rsa-2048.ocsp,acmecerts/unified/live/ec-prime256v1.ocsp
ssl_cert_name=acmecerts/wikiworkshop/live/rsa-2048.chained.crt,acmecerts/wikiworkshop/live/ec-prime256v1.chained.crt ssl_key_name=acmecerts/wikiworkshop/live/rsa-2048.key,acmecerts/wikiworkshop/live/ec-prime256v1.key ssl_ocsp_name=acmecerts/wikiworkshop/live/rsa-2048.ocsp,acmecerts/wikiworkshop/live/ec-prime256v1.ocsp

vgutierrez@cp1081:~$ sudo -i traffic_ctl --run-root=/srv/trafficserver/tls config diff
NOTE: using command line path as RUNROOT
proxy.config.http.server_ports has changed
	Current Value   : 443:ssl 443:ipv6:ssl
	Default Value   : 8080 8080:ipv6
proxy.config.admin.user_id has changed
	Current Value   : trafficserver
	Default Value   : root
proxy.config.http.insert_request_via_str has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.http.insert_client_ip has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.http.response_server_enabled has changed
	Current Value   : 2
	Default Value   : 1
proxy.config.url_remap.pristine_host_hdr has changed
	Current Value   : 1
	Default Value   : 0
proxy.config.disable_configuration_modification has changed
	Current Value   : 1
	Default Value   : 0
proxy.config.net.connections_throttle has changed
	Current Value   : 0
	Default Value   : 30000
proxy.config.net.sock_option_flag_in has changed
	Current Value   : 13
	Default Value   : 5
proxy.config.net.default_inactivity_timeout has changed
	Current Value   : 300
	Default Value   : 86400
proxy.config.http2.max_priority_frames_per_minute has changed
	Current Value   : 0
	Default Value   : 120
proxy.config.http2.accept_no_activity_timeout has changed
	Current Value   : 180
	Default Value   : 120
proxy.config.http2.no_activity_timeout_in has changed
	Current Value   : 180
	Default Value   : 120
proxy.config.http2.stream_error_rate_threshold has changed
	Current Value   : 1.000000
	Default Value   : 0.100000
proxy.config.http.connect_attempts_timeout has changed
	Current Value   : 3
	Default Value   : 30
proxy.config.http.post_connect_attempts_timeout has changed
	Current Value   : 3
	Default Value   : 1800
proxy.config.http.transaction_no_activity_timeout_in has changed
	Current Value   : 200
	Default Value   : 30
proxy.config.http.transaction_no_activity_timeout_out has changed
	Current Value   : 180
	Default Value   : 30
proxy.config.http.accept_no_activity_timeout has changed
	Current Value   : 180
	Default Value   : 120
proxy.config.http.send_100_continue_response has changed
	Current Value   : 1
	Default Value   : 0
proxy.config.http.max_post_size has changed
	Current Value   : 104857600
	Default Value   : 0
proxy.config.http.server_session_sharing.match has changed
	Current Value   : ip
	Default Value   : both
proxy.config.http.auth_server_session_private has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.http.cache.http has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.http.cache.max_open_read_retries has changed
	Current Value   : 50
	Default Value   : -1
proxy.config.http.cache.max_open_write_retries has changed
	Current Value   : 150
	Default Value   : 1
proxy.config.diags.output.status has changed
	Current Value   : O
	Default Value   : L
proxy.config.diags.output.note has changed
	Current Value   : O
	Default Value   : L
proxy.config.diags.output.warning has changed
	Current Value   : O
	Default Value   : L
proxy.config.diags.output.error has changed
	Current Value   : E
	Default Value   : L
proxy.config.diags.output.fatal has changed
	Current Value   : E
	Default Value   : L
proxy.config.diags.output.alert has changed
	Current Value   : E
	Default Value   : L
proxy.config.diags.output.emergency has changed
	Current Value   : E
	Default Value   : L
proxy.config.log.max_space_mb_headroom has changed
	Current Value   : 0
	Default Value   : 1000
proxy.config.log.rolling_enabled has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.log.max_secs_per_buffer has changed
	Current Value   : 0
	Default Value   : 5
proxy.config.log.file_stat_frequency has changed
	Current Value   : 1
	Default Value   : 32
proxy.config.log.log_buffer_size has changed
	Current Value   : 147456
	Default Value   : 9216
proxy.config.log.max_line_size has changed
	Current Value   : 147456
	Default Value   : 9216
proxy.config.ssl.TLSv1 has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.ssl.TLSv1_1 has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.ssl.TLSv1_3 has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.ssl.server.cipher_suite has changed
	Current Value   : -ALL:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA
	Default Value   : ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
proxy.config.ssl.server.groups_list has changed
	Current Value   : X25519:prime256v1
	Default Value   : NULL
proxy.config.ssl.server.cert.path has changed
	Current Value   : /srv/trafficserver/tls/etc/tls
	Default Value   : /etc/trafficserver
proxy.config.ssl.server.private_key.path has changed
	Current Value   : /srv/trafficserver/tls/etc/tls
	Default Value   : /etc/trafficserver
proxy.config.ssl.max_record_size has changed
	Current Value   : 16383
	Default Value   : 0
proxy.config.ssl.server.dhparams_file has changed
	Current Value   : /etc/ssl/dhparam.pem
	Default Value   : NULL
proxy.config.ssl.ocsp.enabled has changed
	Current Value   : 1
	Default Value   : 0
proxy.config.ssl.ocsp.response.path has changed
	Current Value   : /srv/trafficserver/tls/etc/tls
	Default Value   : /etc/trafficserver
proxy.config.ssl.session_cache.timeout has changed
	Current Value   : 7200
	Default Value   : 0
proxy.config.ssl.session_cache.size has changed
	Current Value   : 512000
	Default Value   : 102400
proxy.config.ssl.session_cache.num_buckets has changed
	Current Value   : 32768
	Default Value   : 256
proxy.config.ssl.cert.load_elevated has changed
	Current Value   : 1
	Default Value   : 0
proxy.config.ssl.server.session_ticket.enable has changed
	Current Value   : 0
	Default Value   : 1
proxy.config.ssl.handshake_timeout_in has changed
	Current Value   : 60
	Default Value   : 0
proxy.config.body_factory.template_sets_dir has changed
	Current Value   : /srv/trafficserver/tls/etc/error_template
	Default Value   : body_factory```

jvgutierrez avatar Apr 03 '20 14:04 jvgutierrez

Things have improved a lot by backporting https://github.com/apache/trafficserver/pull/5697, I'm not sure it's the only one, but that's definitely helping:)

jvgutierrez avatar Apr 10 '20 06:04 jvgutierrez

The issue is still there sadly : image

jvgutierrez avatar Apr 10 '20 14:04 jvgutierrez

perf top points to an issue related to HTTP/2:

10.12%  traffic_server  [.] Http2Stream::signal_write_event
10.01%  traffic_server  [.] Http2Stream::update_write_request
 9.98%  traffic_server  [.] Http2ConnectionState::send_a_data_frame

jvgutierrez avatar Apr 11 '20 16:04 jvgutierrez

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.

github-actions[bot] avatar Jun 18 '21 18:06 github-actions[bot]

@jvgutierrez are you still seeing memory leaks? What version of ATS are you running?

bryancall avatar Oct 25 '21 23:10 bryancall