firmware
firmware copied to clipboard
meshtastic
[Bug]: linux-native: Crash on HTTPs request
Category
Other
Hardware
Linux Native
Is this bug report about any UI component firmware like InkHUD or Meshtatic UI (MUI)?
- [ ] Meshtastic UI aka MUI colorTFT
- [ ] InkHUD ePaper
- [ ] OLED slide UI on any display
Firmware Version
Nightly
Description
There have been sporadic crashes when connecting over https to linux-native. Redoing the connect procedure from the web client using HTTPs sooner or later triggers a crash.
With latest stable installed from the Opensuse Raspbian repo, most of the time it happens after a long (300+ packets) stream of node info packets at inital connection using the web client.
In an attempt to debug this further, I compiled native-tft-debug from main (git hash 720add72) which instead resulted in crash very early in the connect process (before any nodeinfo packets were sent).
Running on a Raspberry Pi 3B with a DIY SX1262 hat.
Linux meshtastic 6.12.25+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.12.25-1+rpt1 (2025-04-30) aarch64 GNU/Linux
Relevant log output
On latest beta:
[...]
Jun 10 22:16:38 meshtastic meshtasticd[9822]: INFO | 22:16:38 268 nodeinfo: num=0x336368dc, lastseen=1749525088, id=!336368dc, name=Name
Jun 10 22:16:38 meshtastic meshtasticd[9822]: DEBUG | 22:16:38 268 Send known nodes
Jun 10 22:16:38 meshtastic meshtasticd[9822]: INFO | 22:16:38 268 nodeinfo: num=0xda634ff8, lastseen=1749480225, id=!da634ff8, name=Name
Jun 10 22:16:38 meshtastic meshtasticd[9822]: DEBUG | 22:16:38 268 Send known nodes
Jun 10 22:16:38 meshtastic meshtasticd[9822]: INFO | 22:16:38 268 nodeinfo: num=0xe2e38730, lastseen=1749581614, id=!e2e38730, name=Name
Jun 10 22:16:38 meshtastic meshtasticd[9822]: DEBUG | 22:16:38 268 Send known nodes
Jun 10 22:16:38 meshtastic meshtasticd[9822]: INFO | 22:16:38 268 nodeinfo: num=0x2bafe268, lastseen=1749401817, id=!2bafe268, name=Name
Jun 10 22:16:38 meshtastic meshtasticd[9822]: DEBUG | 22:16:38 268 Send known nodes
Jun 10 22:16:38 meshtastic meshtasticd[9822]: INFO | 22:16:38 268 nodeinfo: num=0x2f9592dc, lastseen=1749441835, id=!2f9592dc, name=Name
Jun 10 22:16:39 meshtastic meshtasticd[9822]: DEBUG | 22:16:38 268 Send known nodes
Jun 10 22:16:39 meshtastic systemd[1]: meshtasticd.service: Main process exited, code=killed, status=11/SEGV
Jun 10 22:16:39 meshtastic systemd[1]: meshtasticd.service: Failed with result 'signal'.
Jun 10 22:16:39 meshtastic systemd[1]: meshtasticd.service: Consumed 2.997s CPU time.
Jun 10 22:16:42 meshtastic systemd[1]: meshtasticd.service: Scheduled restart job, restart counter is at 2.
Jun 10 22:16:42 meshtastic systemd[1]: Stopped meshtasticd.service - Meshtastic Native Daemon.
Jun 10 22:16:42 meshtastic systemd[1]: meshtasticd.service: Consumed 2.997s CPU time.
Jun 10 22:16:42 meshtastic systemd[1]: Started meshtasticd.service - Meshtastic Native Daemon.
Jun 10 22:16:42 meshtastic meshtasticd[9836]: Portduino is starting, VFS root at /root/.portduino/default
Jun 10 22:16:42 meshtastic meshtasticd[9836]: Set up Meshtastic on Portduino...
Jun 10 22:16:42 meshtastic meshtasticd[9836]: Using /etc/meshtasticd/config.yaml as config file
Jun 10 22:16:42 meshtastic meshtasticd[9836]: Also using "/etc/meshtasticd/config.d/lora-protoboard.yaml" as additional config file
Jun 10 22:16:42 meshtastic meshtasticd[9836]: MAC ADDRESS: B8:27:EB:35:38:A2
Jun 10 22:16:42 meshtastic meshtasticd[9836]: DEBUG | ??:??:?? 0 Upgrade time to quality NTP
Jun 10 22:16:42 meshtastic meshtasticd[9836]: DEBUG | 22:16:42 0 Read RTC time as 1749586602
Jun 10 22:16:42 meshtastic meshtasticd[9836]: INFO | 22:16:42 0
Jun 10 22:16:42 meshtastic meshtasticd[9836]:
Jun 10 22:16:42 meshtastic meshtasticd[9836]: //\ E S H T /\ S T / C
Jun 10 22:16:42 meshtastic meshtasticd[9836]:
[...]
native-tft-debug (git hash 720add72):
(gdb) run
Starting program: /home/mesh/meshtastic-firmware/firmware/.pio/build/native-tft-debug/program
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Portduino is starting, VFS root at /root/.portduino/default
Set up Meshtastic on Portduino...
Using /etc/meshtasticd/config.yaml as config file
Also using "/etc/meshtasticd/config.d/lora-protoboard.yaml" as additional config file
MAC ADDRESS: B8:27:EB:35:38:A2
DEBUG | ??:??:?? 0 Upgrade time to quality NTP
DEBUG | 18:42:23 0 Read RTC time as 1749573743
INFO | 18:42:23 0
//\ E S H T /\ S T / C
DEBUG | 18:42:23 0 Filesystem files:
DEBUG | 18:42:23 0 /.. (0 Bytes)
DEBUG | 18:42:23 0 /prefs/.. (0 Bytes)
DEBUG | 18:42:23 0 /prefs/nodes.proto (43911 Bytes)
DEBUG | 18:42:23 0 /prefs/. (0 Bytes)
DEBUG | 18:42:23 0 /prefs/device.proto (199 Bytes)
DEBUG | 18:42:23 0 /prefs/module.proto (124 Bytes)
DEBUG | 18:42:23 0 /prefs/channels.proto (159 Bytes)
DEBUG | 18:42:23 0 /prefs/config.proto (236 Bytes)
DEBUG | 18:42:23 0 /. (0 Bytes)
INFO | 18:42:23 0 No I2C device configured, Skip
INFO | 18:42:23 0 Running without TFT display!
INFO | 18:42:23 0 No I2C devices found
DEBUG | 18:42:23 0 acc_info = 0
INFO | 18:42:23 0 S:B:37,2.6.11.720add72
INFO | 18:42:23 0 Init NodeDB
INFO | 18:42:23 0 Load /prefs/nodes.proto
INFO | 18:42:23 0 Loaded /prefs/nodes.proto successfully
INFO | 18:42:23 0 Loaded saved nodedatabase version 24, with nodes count: 375
INFO | 18:42:23 0 Load /prefs/device.proto
INFO | 18:42:23 0 Loaded /prefs/device.proto successfully
INFO | 18:42:23 0 Loaded saved devicestate version 24
INFO | 18:42:23 0 Load /prefs/config.proto
INFO | 18:42:23 0 Loaded /prefs/config.proto successfully
INFO | 18:42:23 0 Loaded saved config version 24
INFO | 18:42:23 0 Load /prefs/module.proto
INFO | 18:42:23 0 Loaded /prefs/module.proto successfully
INFO | 18:42:23 0 Loaded saved moduleConfig version 24
INFO | 18:42:23 0 Load /prefs/channels.proto
INFO | 18:42:23 0 Loaded /prefs/channels.proto successfully
INFO | 18:42:23 0 Loaded saved channelFile version 24
/root/.portduino/default/prefs/uiconfig.proto does not exist
ERROR | 18:42:23 0 Could not open / read /prefs/uiconfig.proto
DEBUG | 18:42:23 0 cleanupMeshDB purged 0 entries
DEBUG | 18:42:23 0 Use nodenum 0xeb3538a2
DEBUG | 18:42:23 0 Expand short PSK #1
INFO | 18:42:23 0 Wanted region 3, using EU_868
DEBUG | 18:42:23 0 Save to disk 0
DEBUG | 18:42:23 0 Use compiled/slipstreamed tzplaceholder
DEBUG | 18:42:23 0 Saved TZ: CET-1CEST,M3.5.0,M10.5.0/3
DEBUG | 18:42:23 0 Set Timezone to CET-1CEST,M3.5.0,M10.5.0/3
DEBUG | 18:42:23 0 Read RTC time as 1749573743
DEBUG | 18:42:23 0 Start multicast thread
[New Thread 0x7feecfee00 (LWP 9447)]
DEBUG | 18:42:23 0 UDP Listening
INFO | 18:42:23 0 External Notification Module Disabled
DEBUG | 18:42:23 0 Activate sx1262 radio on SPI port /dev/spidev0.0
DEBUG | 18:42:23 0 SX126xInterface(cs=21, irq=16, rst=18, busy=20)
DEBUG | 18:42:23 0 SX126X_DIO3_TCXO_VOLTAGE defined, using DIO3 as TCXO reference voltage at 1.800000 V
INFO | 18:42:23 0 Start meshradio init
INFO | 18:42:23 0 Radio freq=869.525, config.lora.frequency_offset=0.000
INFO | 18:42:23 0 Set radio: region=EU_868, name=ph, config=0, ch=0, power=27
INFO | 18:42:23 0 myRegion->freqStart -> myRegion->freqEnd: 869.400024 -> 869.650024 (0.250000 MHz)
INFO | 18:42:23 0 numChannels: 1 x 250.000kHz
INFO | 18:42:23 0 channel_num: 1
INFO | 18:42:23 0 frequency: 869.525024
INFO | 18:42:23 0 Slot time: 28 msec
INFO | 18:42:23 0 Final Tx power: 27 dBm
INFO | 18:42:23 0 SX126x init result 0
INFO | 18:42:23 0 Frequency set to 869.525024
INFO | 18:42:23 0 Bandwidth set to 250.000000
INFO | 18:42:23 0 Power output set to 22
DEBUG | 18:42:23 0 Current limit set to 140.000000
DEBUG | 18:42:23 0 Current limit set result 0
DEBUG | 18:42:23 0 Set DIO2 as not RF switch, result: 0
DEBUG | 18:42:23 0 Use MCU pin 12 as RXEN and pin 13 as TXEN to control RF switching
INFO | 18:42:23 0 Set RX gain to boosted mode; result: 0
INFO | 18:42:23 0 sx1262 init success
DEBUG | 18:42:23 0 Init MQTT
INFO | 18:42:23 0 MQTT server on a private IP
INFO | 18:42:23 0 Use webserver port from yaml config 443
INFO | 18:42:23 0 Webserver started
[New Thread 0x7fedefee00 (LWP 9448)]
INFO | 18:42:23 0 Web Server framework started on port: 443
INFO | 18:42:23 0 Web Server root /usr/share/meshtasticd/web
INFO | 18:42:23 0 API server listen on TCP port 4403
DEBUG | 18:42:23 0 LoRA bitrate = 118.394310 bytes / sec
INFO | 18:42:23 0 PowerFSM init, USB power=1
DEBUG | 18:42:23 0 State: BOOT
INFO | 18:42:23 0 [RangeTest] Range Test Module - Disabled
INFO | 18:42:23 0 [mqtt] Connecting directly to MQTT server IP, port: 1883, username: USER, password: PASSWORD
INFO | 18:42:23 0 [mqtt] MQTT connected
INFO | 18:42:23 0 [mqtt] MQTT server on a private IP
[New Thread 0x7fed6eee00 (LWP 9449)]
DEBUG | 18:42:32 9 handleAPIv1ToRadio web -> radio
DEBUG | 18:42:32 9 handleAPIv1ToRadio web -> radio
=================================================================
==9445==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x007ff2510950 at pc 0x005555829c64 bp 0x007fed6edd30 sp 0x007fed6edd48
READ of size 512 at 0x007ff2510950 thread T3
#0 0x5555829c60 in handleAPIv1ToRadio(_u_request const*, _u_response*, void*) src/mesh/raspihttp/PiWebServer.cpp:244
#1 0x7ff7433840 (/lib/aarch64-linux-gnu/libulfius.so.2.7+0x13840)
#2 0x7ff6605e24 (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0x5e24)
#3 0x7ff6607b2c (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0x7b2c)
#4 0x7ff660a2cc (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0xa2cc)
#5 0x7ff660ddb8 (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0xddb8)
#6 0x7ff6a7ee9c in start_thread nptl/pthread_create.c:442
#7 0x7ff6ae7b18 in thread_start ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
0x007ff2510956 is located 0 bytes to the right of 6-byte region [0x007ff2510950,0x007ff2510956)
allocated by thread T3 here:
#0 0x7ff78fa794 in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:85
#1 0x7ff7433a44 (/lib/aarch64-linux-gnu/libulfius.so.2.7+0x13a44)
#2 0x7ff66089bc (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0x89bc)
#3 0x7ff660a2cc (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0xa2cc)
#4 0x7ff660ddb8 (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0xddb8)
#5 0x7ff6a7ee9c in start_thread nptl/pthread_create.c:442
#6 0x7ff6ae7b18 in thread_start ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
Thread T3 created by T2 here:
#0 0x7ff789a234 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207
#1 0x7ff6616ec0 (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0x16ec0)
#2 0x7ff6616fac (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0x16fac)
#3 0x7ff660c4d0 (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0xc4d0)
#4 0x7ff660d374 (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0xd374)
#5 0x7ff660e4a4 (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0xe4a4)
#6 0x7ff660f980 (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0xf980)
#7 0x7ff660fff4 (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0xfff4)
#8 0x7ff6a7ee9c in start_thread nptl/pthread_create.c:442
#9 0x7ff6ae7b18 in thread_start ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
Thread T2 created by T0 here:
#0 0x7ff789a234 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:207
#1 0x7ff6616ec0 (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0x16ec0)
#2 0x7ff6616fac (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0x16fac)
#3 0x7ff661138c in MHD_start_daemon_va (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0x1138c)
#4 0x7ff6612290 in MHD_start_daemon (/lib/aarch64-linux-gnu/libmicrohttpd.so.12+0x12290)
#5 0x7ff7434ea0 in ulfius_start_secure_ca_trust_framework (/lib/aarch64-linux-gnu/libulfius.so.2.7+0x14ea0)
#6 0x555582bb8c in PiWebServerThread::PiWebServerThread() src/mesh/raspihttp/PiWebServer.cpp:514
#7 0x55557b52b0 in setup src/main.cpp:1268
#8 0x5555a7c444 in main /home/mesh/.platformio/packages/framework-portduino/cores/portduino/main.cpp:166
#9 0x7ff6a2773c in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#10 0x7ff6a27814 in __libc_start_main_impl ../csu/libc-start.c:360
#11 0x55556b63ac in _start (/home/mesh/meshtastic-firmware/firmware/.pio/build/native-tft-debug/program+0x1663ac)
SUMMARY: AddressSanitizer: heap-buffer-overflow src/mesh/raspihttp/PiWebServer.cpp:244 in handleAPIv1ToRadio(_u_request const*, _u_response*, void*)
Shadow bytes around the buggy address:
0x001ffe4a20d0: fa fa fd fd fa fa fd fa fa fa fd fd fa fa fd fa
0x001ffe4a20e0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fa
0x001ffe4a20f0: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
0x001ffe4a2100: fa fa fd fd fa fa fd fa fa fa fd fa fa fa fd fa
0x001ffe4a2110: fa fa fd fa fa fa fd fa fa fa fd fd fa fa fd fd
=>0x001ffe4a2120: fa fa fd fa fa fa fd fa fa fa[06]fa fa fa 05 fa
0x001ffe4a2130: fa fa 00 01 fa fa 00 04 fa fa 00 07 fa fa 06 fa
0x001ffe4a2140: fa fa 00 07 fa fa 07 fa fa fa 00 07 fa fa 00 04
0x001ffe4a2150: fa fa 00 03 fa fa 03 fa fa fa 04 fa fa fa 07 fa
0x001ffe4a2160: fa fa 03 fa fa fa 00 07 fa fa 00 05 fa fa 00 fa
0x001ffe4a2170: fa fa 00 00 fa fa 00 00 fa fa 00 00 fa fa 05 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==9445==ABORTING
[Thread 0x7fed6eee00 (LWP 9449) exited]
[Thread 0x7feecfee00 (LWP 9447) exited]
[Thread 0x7ff77bd040 (LWP 9445) exited]
[Thread 0x7fedefee00 (LWP 9448) exited]
[New process 9445]
[Inferior 1 (process 9445) exited with code 01]
