pixie
pixie copied to clipboard
[tcp_stats] Use kfuncs to probe the tcp_sendmsg function
Summary: Adds kfunc support to BCC wrapper and utilizes it to probe the tcp_sendmsg function from the tcp stats connector.
We had seen a kprobe attachment conflict when the socket tracer and tcp stats connectors were enabled in Stirling due to both connectors trying to deploy kprobes for tcp_sendmsg. This PR modifies the tcp stats connector to instead deploy a kfunc probe to avoid the conflict.
Relevant Issues: #2055
Type of change: /kind bug
Test Plan: Skaffolded these changes on a dev cluster with socket tracer and tcp stats enabled and verified the existing tcp stats BPF test passes on kernel version 6.8.0-1018-gcp
Relevant parts of the pem logs from skaffolding:
I20250106 22:11:24.698770 18078 source_connector.cc:35] Initializing source connector: socket_tracer
I20250106 22:11:24.698838 18078 socket_trace_connector.cc:475] Kernel version greater than V5.1 detected (6.1.112), raised loop limit to 882 and chunk limit to 84
I20250106 22:11:24.698889 18078 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.112+
I20250106 22:11:24.698920 18078 linux_headers.cc:353] Detected kernel release (uname -r): 6.1.112+
I20250106 22:11:24.699201 18078 linux_headers.cc:369] Not Found : Could not find 'source' or 'build' under /lib/modules/6.1.112+.
I20250106 22:11:24.699244 18078 linux_headers.cc:215] Looking for host Linux headers at /host/lib/modules/6.1.112+/build.
I20250106 22:11:24.699276 18078 linux_headers.cc:372] Not Found : Did not find the host headers at path: /host/lib/modules/6.1.112+/build, No such file or directory.
I20250106 22:11:24.699309 18078 linux_headers.cc:314] Attempting to install packaged headers.
W20250106 22:11:24.699962 18078 linux_headers.cc:287] Ignoring /px/linux-headers-x86_64-4.19.271.tar.gz since it does not conform to the naming convention
W20250106 22:11:24.700018 18078 linux_headers.cc:287] Ignoring /px/linux-headers-x86_64-4.14.304.tar.gz since it does not conform to the naming convention
I20250106 22:11:24.700129 18078 linux_headers.cc:320] Using packaged header: /px/linux-headers-x86_64-6.1.8.tar.gz
I20250106 22:11:25.539538 18078 linux_headers.cc:57] Overriding linux version code to 393584
I20250106 22:11:25.540490 18078 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.112+
I20250106 22:11:25.540580 18078 linux_headers.cc:99] Found kernel config at: /proc/config.gz.
I20250106 22:11:25.558293 18078 linux_headers.cc:345] Successfully installed packaged copy of headers at /lib/modules/6.1.112+/build
I20250106 22:11:25.558351 18078 bcc_wrapper.cc:94] Resolving task_struct offsets.
I20250106 22:11:25.694319 18094 bcc_wrapper.cc:166] Initializing BPF program ...
I20250106 22:11:26.983464 18094 scoped_timer.h:48] Timer(init_bpf_program) : 1.29 s
I20250106 22:11:27.003376 18078 bcc_wrapper.cc:166] Initializing BPF program ...
I20250106 22:11:28.365095 18078 scoped_timer.h:48] Timer(init_bpf_program) : 1.36 s
I20250106 22:11:28.383139 18078 bcc_wrapper.cc:97] Successfully resolved task_struct offsets: {real_start_time=1832, group_leader=1584, exit_code=1416}
I20250106 22:11:28.383253 18078 bcc_wrapper.cc:166] Initializing BPF program ...
I20250106 22:12:26.397286 18078 scoped_timer.h:48] Timer(init_bpf_program) : 58.01 s
I20250106 22:12:29.713171 18078 socket_trace_connector.cc:500] Number of kprobes deployed = 46
I20250106 22:12:29.713227 18078 socket_trace_connector.cc:501] Probes successfully deployed.
I20250106 22:12:29.713286 18078 socket_trace_connector.cc:415] Initializing perf buffers with ncpus=4 and scaling_factor=0.75
I20250106 22:12:29.713353 18078 socket_trace_connector.cc:404] Total perf buffer usage for kData buffers across all cpus: 266587120
I20250106 22:12:29.713382 18078 socket_trace_connector.cc:404] Total perf buffer usage for kControl buffers across all cpus: 6606028
I20250106 22:12:29.821144 18078 socket_trace_connector.cc:505] Number of perf buffers opened = 8
I20250106 22:12:30.109012 18078 stirling.cc:438] Adding info class: [socket_tracer/conn_stats]
I20250106 22:12:30.109094 18078 stirling.cc:438] Adding info class: [socket_tracer/http_events]
I20250106 22:12:30.109122 18078 stirling.cc:438] Adding info class: [socket_tracer/mysql_events]
I20250106 22:12:30.109141 18078 stirling.cc:438] Adding info class: [socket_tracer/cql_events]
I20250106 22:12:30.109160 18078 stirling.cc:438] Adding info class: [socket_tracer/pgsql_events]
I20250106 22:12:30.109181 18078 stirling.cc:438] Adding info class: [socket_tracer/dns_events]
I20250106 22:12:30.109203 18078 stirling.cc:438] Adding info class: [socket_tracer/redis_events]
I20250106 22:12:30.109225 18078 stirling.cc:438] Adding info class: [socket_tracer/nats_events.beta]
I20250106 22:12:30.109246 18078 stirling.cc:438] Adding info class: [socket_tracer/kafka_events.beta]
I20250106 22:12:30.109282 18078 stirling.cc:438] Adding info class: [socket_tracer/mux_events]
I20250106 22:12:30.109303 18078 stirling.cc:438] Adding info class: [socket_tracer/amqp_events]
I20250106 22:12:30.109321 18078 stirling.cc:438] Adding info class: [socket_tracer/mongodb_events]
I20250106 22:12:30.109455 18078 source_connector.cc:35] Initializing source connector: tcp_stats
I20250106 22:12:30.109503 18078 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.1.112+
I20250106 22:12:30.109681 18078 linux_headers.cc:353] Detected kernel release (uname -r): 6.1.112+
I20250106 22:12:30.109760 18078 linux_headers.cc:207] Using Linux headers from: /lib/modules/6.1.112+/build.
I20250106 22:12:30.109797 18078 bcc_wrapper.cc:90] Returning the previously resolved TaskStructOffsets object
I20250106 22:12:30.109912 18078 bcc_wrapper.cc:166] Initializing BPF program ...
I20250106 22:12:34.116115 18078 scoped_timer.h:48] Timer(init_bpf_program) : 4.01 s
I20250106 22:12:34.294086 18078 tcp_stats_connector.cc:92] Successfully deployed 2 kprobes.
I20250106 22:12:34.294157 18078 tcp_stats_connector.cc:93] Successfully deployed 2 kfunc probes.
I20250106 22:12:34.294176 18078 stirling.cc:438] Adding info class: [tcp_stats/tcp_stats_events]