CESNET
Memory leak detected in valgrind when running libyang
Version: Libyang 2.1.80
Hi There,
We have an application and it has been found recently that a process in that application has been increasing in memory. Through some binary searching I was able to narrow it down to a change that was made involving libyang.
We have a typescript project that makes use of napi-rs to call rust code that natively runs libyang
I managed to run valgrind on our application running libyang and a definite memory leak was detected
==23236== LEAK SUMMARY: ==23236== definitely lost: 2,048 bytes in 1 blocks ==23236== indirectly lost: 0 bytes in 0 blocks ==23236== possibly lost: 857,350 bytes in 7,307 blocks ==23236== still reachable: 384 bytes in 2 blocks ==23236== of which reachable via heuristic: ==23236== newarray : 150,032 bytes in 1,044 blocks ==23236== suppressed: 0 bytes in 0 blocks ==23236== Reachable blocks (those to which a pointer was found) are not shown. ==23236== To see them, rerun with: --leak-check=full --show-leak-kinds=all
The leak seems to be coming from here
==23236== 2,048 bytes in 1 blocks are definitely lost in loss record 1,061 of 1,125 ==23236== at 0x483AD7B: realloc (vg_replace_malloc.c:834) ==23236== by 0x485F58C: ly_realloc (in /usr/lib/x86_64-linux-gnu/libyang.so.2) ==23236== by 0x48907C7: ly_write_ (in /usr/lib/x86_64-linux-gnu/libyang.so.2) ==23236== by 0x4892B7F: json_print_string.part.0 (in /usr/lib/x86_64-linux-gnu/libyang.so.2) ==23236== by 0x4892C68: json_print_value (in /usr/lib/x86_64-linux-gnu/libyang.so.2) ==23236== by 0x4893C3F: json_print_node.part.0 (in /usr/lib/x86_64-linux-gnu/libyang.so.2) ==23236== by 0x48946FA: json_print_inner (in /usr/lib/x86_64-linux-gnu/libyang.so.2) ==23236== by 0x4894212: json_print_node.part.0 (in /usr/lib/x86_64-linux-gnu/libyang.so.2) ==23236== by 0x48946FA: json_print_inner (in /usr/lib/x86_64-linux-gnu/libyang.so.2) ==23236== by 0x4894212: json_print_node.part.0 (in /usr/lib/x86_64-linux-gnu/libyang.so.2) ==23236== by 0x48946FA: json_print_inner (in /usr/lib/x86_64-linux-gnu/libyang.so.2) ==23236== by 0x4893C72: json_print_node.part.0 (in /usr/lib/x86_64-linux-gnu/libyang.so.2)
If I run the application multiple times then the memory lost will increase by number of runs * 2Kb
Kind regards
Please rerun the valgrind command with --num-callers=30 so that the full stack trace can be seen, thanks.
Thankss for the reply.
I tried updating to the latest version of libyang 2.1 (2.1.148) to see if that changed any behaviour. The following stack traces are for that version.
I have run the added the command and it seems to happen if we add the WITH_SIBLINGS flag when printing
==29002== at 0x483AD7B: realloc (vg_replace_malloc.c:834) ==29002== by 0x485FA7C: ly_realloc (in /usr/local/lib/libyang.so.2.46.3) ==29002== by 0x4892CC7: ly_write_ (in /usr/local/lib/libyang.so.2.46.3) ==29002== by 0x489520F: json_print_string.part.0 (in /usr/local/lib/libyang.so.2.46.3) ==29002== by 0x489528B: json_print_value (in /usr/local/lib/libyang.so.2.46.3) ==29002== by 0x48964B1: json_print_node.part.0 (in /usr/local/lib/libyang.so.2.46.3) ==29002== by 0x4896F92: json_print_inner (in /usr/local/lib/libyang.so.2.46.3) ==29002== by 0x4896B42: json_print_node.part.0 (in /usr/local/lib/libyang.so.2.46.3) ==29002== by 0x4896F92: json_print_inner (in /usr/local/lib/libyang.so.2.46.3) ==29002== by 0x4896B42: json_print_node.part.0 (in /usr/local/lib/libyang.so.2.46.3) ==29002== by 0x4896F92: json_print_inner (in /usr/local/lib/libyang.so.2.46.3) ==29002== by 0x48964EA: json_print_node.part.0 (in /usr/local/lib/libyang.so.2.46.3) ==29002== by 0x48973EA: json_print_data (in /usr/local/lib/libyang.so.2.46.3) ==29002== by 0x4893772: lyd_print_mem (in /usr/local/lib/libyang.so.2.46.3) ==29002== by 0x119FBC: yang3::data::Data::print_string (data.rs:349)
==32264== LEAK SUMMARY: ==32264== definitely lost: 2,048 bytes in 1 blocks ==32264== indirectly lost: 0 bytes in 0 blocks ==32264== possibly lost: 922,950 bytes in 7,309 blocks ==32264== still reachable: 448 bytes in 2 blocks ==32264== of which reachable via heuristic: ==32264== newarray : 150,032 bytes in 1,044 blocks ==32264== suppressed: 0 bytes in 0 blocks ==32264== Reachable blocks (those to which a pointer was found) are not shown. ==32264== To see them, rerun with: --leak-check=full --show-leak-kinds=all
If I don't use the WITH_SIBLINGS flag the memory leak is not nearly as bad
==30633== LEAK SUMMARY: ==30633== definitely lost: 6 bytes in 1 blocks ==30633== indirectly lost: 0 bytes in 0 blocks ==30633== possibly lost: 922,950 bytes in 7,309 blocks ==30633== still reachable: 448 bytes in 2 blocks ==30633== of which reachable via heuristic: ==30633== newarray : 150,032 bytes in 1,044 blocks ==30633== suppressed: 0 bytes in 0 blocks ==30633== Reachable blocks (those to which a pointer was found) are not shown. ==30633== To see them, rerun with: --leak-check=full --show-leak-kinds=all
As an added bit of information I have possibly found the leak.
The allocation is happening here
out->method.mem.buf = ly_realloc(*out->method.mem.buf, new_mem_size);
if (!*out->method.mem.buf)
{
out->method.mem.len = 0;
out->method.mem.size = 0;
LOGMEM(NULL);
return LY_EMEM;
}
so memory is being allocated to out->method.mem.buf
lyd_print_mem(char **strp, const struct lyd_node *root, LYD_FORMAT format, uint32_t options)
{
LY_ERR ret;
struct ly_out *out;
LY_CHECK_ARG_RET(NULL, strp, LY_EINVAL);
/* init */
*strp = NULL;
LY_CHECK_RET(ly_out_new_memory(strp, 0, &out));
ret = lyd_print_(out, root, format, options);
ly_out_free(out, NULL, 0);
return ret;
}
lyd_print_mem calls ly_out_free with destroy of 0 so the buffer will never be cleared
I have created a pull request https://github.com/CESNET/libyang/pull/2336
Changing the 0 to a 1 and I get
=85529== LEAK SUMMARY: ==85529== definitely lost: 0 bytes in 0 blocks ==85529== indirectly lost: 0 bytes in 0 blocks ==85529== possibly lost: 931,248 bytes in 7,317 blocks ==85529== still reachable: 448 bytes in 2 blocks ==85529== of which reachable via heuristic: ==85529== newarray : 152,392 bytes in 1,046 blocks ==85529== suppressed: 0 bytes in 0 blocks ==85529== Reachable blocks (those to which a pointer was found) are not shown. ==85529== To see them, rerun with: --leak-check=full --show-leak-kinds=all
Changing it back to a 0 and I get
==88651== LEAK SUMMARY: ==88651== definitely lost: 2,048 bytes in 1 blocks ==88651== indirectly lost: 0 bytes in 0 blocks ==88651== possibly lost: 931,248 bytes in 7,317 blocks ==88651== still reachable: 448 bytes in 2 blocks ==88651== of which reachable via heuristic: ==88651== newarray : 152,392 bytes in 1,046 blocks ==88651== suppressed: 0 bytes in 0 blocks ==88651== Reachable blocks (those to which a pointer was found) are not shown. ==88651== To see them, rerun with: --leak-check=full **--show-leak-kinds=all
There is no leak, you must free the output string yourself. It cannot be freed right after printing, you would then get a freed string in your application.
Sorry my mistake. Thank you for the reply. It seems that the actual culprit would be the yang3 package making use of the libyang library.
fn print_string(
&self,
format: DataFormat,
options: DataPrinterFlags,
) -> Result<String> {
let mut cstr = std::ptr::null_mut();
let cstr_ptr = &mut cstr;
let ret = unsafe {
ffi::lyd_print_mem(
cstr_ptr,
self.raw(),
format as u32,
options.bits(),
)
};
if ret != ffi::LY_ERR::LY_SUCCESS {
return Err(Error::new(self.context()));
}
Ok(char_ptr_to_string(cstr))
}
It does not seem to free the cstr